== conf-sanity test 69: replace an OST with the same index ========================================================== 19:35:17 (1781220917) start mds service on oleg135-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 543 (time 1781221001.66 total 10.01 last 54.26) - open/close 1191 (time 1781221011.66 total 20.01 last 64.78) - open/close 1946 (time 1781221021.67 total 30.02 last 75.42) - open/close 2723 (time 1781221031.68 total 40.02 last 77.67) - open/close 3594 (time 1781221041.68 total 50.03 last 87.06) - open/close 4371 (time 1781221051.68 total 60.03 last 77.69) - open/close 5281 (time 1781221061.69 total 70.04 last 90.94) - open/close 6168 (time 1781221071.70 total 80.04 last 88.63) - open/close 6916 (time 1781221081.70 total 90.05 last 74.79) - open/close 7728 (time 1781221091.71 total 100.06 last 81.12) - open/close 8480 (time 1781221101.71 total 110.06 last 75.19) - open/close 9291 (time 1781221111.72 total 120.06 last 81.05) total: 10000 open/close in 128.96 seconds: 77.54 ops/second - unlinked 0 (time 1781221124 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 812 (time 1781221222.34 total 10.01 last 81.16) - open/close 1611 (time 1781221232.35 total 20.02 last 79.80) - open/close 2341 (time 1781221242.39 total 30.06 last 72.72) - open/close 3133 (time 1781221252.39 total 40.06 last 79.20) - open/close 3974 (time 1781221262.41 total 50.08 last 83.94) - open/close 4872 (time 1781221272.41 total 60.08 last 89.79) - open/close 5668 (time 1781221282.43 total 70.09 last 79.48) - open/close 6442 (time 1781221292.44 total 80.11 last 77.27) - open/close 7128 (time 1781221302.45 total 90.11 last 68.57) - open/close 7899 (time 1781221312.45 total 100.11 last 77.09) - open/close 8735 (time 1781221322.47 total 110.13 last 83.46) - open/close 9496 (time 1781221332.47 total 120.14 last 76.05) total: 10000 open/close in 126.52 seconds: 79.04 ops/second - unlinked 0 (time 1781221343 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 662 (time 1781221441.17 total 10.00 last 66.18) - open/close 1140 (time 1781221451.19 total 20.03 last 47.68) - open/close 1688 (time 1781221461.19 total 30.03 last 54.79) - open/close 2450 (time 1781221471.22 total 40.05 last 76.03) - open/close 3067 (time 1781221481.22 total 50.05 last 61.69) - open/close 3855 (time 1781221491.23 total 60.06 last 78.73) - open/close 4666 (time 1781221501.23 total 70.07 last 81.07) - open/close 5444 (time 1781221511.23 total 80.07 last 77.79) - open/close 6237 (time 1781221521.23 total 90.07 last 79.29) - open/close 7069 (time 1781221531.24 total 100.08 last 83.14) - open/close 7927 (time 1781221541.25 total 110.09 last 85.72) - open/close 8728 (time 1781221551.25 total 120.09 last 80.09) - open/close 9554 (time 1781221561.25 total 130.09 last 82.58) total: 10000 open/close in 135.43 seconds: 73.84 ops/second - unlinked 0 (time 1781221570 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 834 (time 1781221670.37 total 10.01 last 83.28) - open/close 1636 (time 1781221680.38 total 20.02 last 80.17) - open/close 2458 (time 1781221690.39 total 30.03 last 82.08) - open/close 3211 (time 1781221700.40 total 40.04 last 75.25) - open/close 4046 (time 1781221710.42 total 50.06 last 83.34) - open/close 4887 (time 1781221720.42 total 60.06 last 84.07) - open/close 5628 (time 1781221730.43 total 70.07 last 74.07) - open/close 6228 (time 1781221740.43 total 80.07 last 60.00) - open/close 7003 (time 1781221750.43 total 90.07 last 77.49) - open/close 7809 (time 1781221760.43 total 100.07 last 80.56) - open/close 8687 (time 1781221770.44 total 110.08 last 87.75) - open/close 9415 (time 1781221780.44 total 120.08 last 72.77) total: 10000 open/close in 128.00 seconds: 78.12 ops/second - unlinked 0 (time 1781221793 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 854 (time 1781221895.37 total 10.00 last 85.37) - open/close 1607 (time 1781221905.38 total 20.01 last 75.28) - open/close 2332 (time 1781221915.39 total 30.02 last 72.43) - open/close 3094 (time 1781221925.39 total 40.02 last 76.14) - open/close 3973 (time 1781221935.40 total 50.03 last 87.84) - open/close 4735 (time 1781221945.40 total 60.03 last 76.18) - open/close 5376 (time 1781221955.42 total 70.05 last 64.03) - open/close 5870 (time 1781221965.42 total 80.05 last 49.39) - open/close 6665 (time 1781221975.42 total 90.05 last 79.49) - open/close 7389 (time 1781221985.42 total 100.05 last 72.36) - open/close 8018 (time 1781221995.42 total 110.05 last 62.90) - open/close 8826 (time 1781222005.43 total 120.06 last 80.76) - open/close 9523 (time 1781222015.44 total 130.07 last 69.62) total: 10000 open/close in 135.87 seconds: 73.60 ops/second - unlinked 0 (time 1781222025 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 541 (time 1781222120.99 total 10.01 last 54.06) - open/close 1265 (time 1781222131.00 total 20.01 last 72.38) - open/close 1906 (time 1781222141.00 total 30.01 last 64.08) - open/close 2504 (time 1781222151.00 total 40.02 last 59.76) - open/close 3044 (time 1781222161.02 total 50.03 last 53.94) - open/close 3532 (time 1781222171.02 total 60.04 last 48.77) - open/close 4064 (time 1781222181.04 total 70.06 last 53.08) - open/close 4611 (time 1781222191.05 total 80.07 last 54.66) - open/close 5344 (time 1781222201.06 total 90.07 last 73.24) - open/close 6160 (time 1781222211.06 total 100.08 last 81.57) - open/close 6884 (time 1781222221.07 total 110.08 last 72.38) - open/close 7696 (time 1781222231.07 total 120.09 last 81.14) - open/close 8400 (time 1781222241.08 total 130.10 last 70.35) - open/close 9133 (time 1781222251.08 total 140.10 last 73.28) - open/close 9895 (time 1781222261.08 total 150.10 last 76.20) total: 10000 open/close in 151.12 seconds: 66.17 ops/second - unlinked 0 (time 1781222266 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 614 (time 1781222366.77 total 10.01 last 61.36) - open/close 1302 (time 1781222376.78 total 20.02 last 68.72) - open/close 1916 (time 1781222386.80 total 30.03 last 61.30) - open/close 2548 (time 1781222396.81 total 40.04 last 63.15) - open/close 3393 (time 1781222406.81 total 50.05 last 84.47) - open/close 4022 (time 1781222416.81 total 60.05 last 62.88) - open/close 4690 (time 1781222426.81 total 70.05 last 66.78) - open/close 5275 (time 1781222436.82 total 80.06 last 58.45) - open/close 5876 (time 1781222446.83 total 90.06 last 60.08) - open/close 6665 (time 1781222456.83 total 100.06 last 78.88) - open/close 7262 (time 1781222466.83 total 110.07 last 59.68) - open/close 7945 (time 1781222476.84 total 120.08 last 68.22) - open/close 8596 (time 1781222486.86 total 130.09 last 65.01) - open/close 9358 (time 1781222496.86 total 140.10 last 76.15) total: 10000 open/close in 148.68 seconds: 67.26 ops/second - unlinked 0 (time 1781222509 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 738 (time 1781222607.72 total 10.01 last 73.69) - open/close 1511 (time 1781222617.72 total 20.02 last 77.26) - open/close 2266 (time 1781222627.72 total 30.02 last 75.49) - open/close 2939 (time 1781222637.73 total 40.03 last 67.25) - open/close 3679 (time 1781222647.74 total 50.04 last 73.92) - open/close 4315 (time 1781222657.75 total 60.05 last 63.55) - open/close 5013 (time 1781222667.76 total 70.06 last 69.72) - open/close 5825 (time 1781222677.77 total 80.06 last 81.14) - open/close 6613 (time 1781222687.78 total 90.07 last 78.74) - open/close 7405 (time 1781222697.80 total 100.10 last 78.98) - open/close 8122 (time 1781222707.81 total 110.10 last 71.67) - open/close 8755 (time 1781222717.81 total 120.11 last 63.26) - open/close 9412 (time 1781222727.82 total 130.12 last 65.66) total: 10000 open/close in 138.11 seconds: 72.41 ops/second - unlinked 0 (time 1781222740 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 751 (time 1781222836.27 total 10.01 last 75.03) - open/close 1501 (time 1781222846.27 total 20.02 last 74.95) - open/close 2214 (time 1781222856.29 total 30.03 last 71.21) - open/close 3001 (time 1781222866.30 total 40.04 last 78.63) - open/close 3698 (time 1781222876.33 total 50.07 last 69.49) - open/close 4352 (time 1781222886.33 total 60.08 last 65.36) - open/close 5088 (time 1781222896.33 total 70.08 last 73.58) - open/close 6139 (time 1781222906.35 total 80.09 last 104.98) - open/close 6820 (time 1781222916.35 total 90.10 last 68.04) - open/close 7479 (time 1781222926.39 total 100.14 last 65.65) - open/close 8285 (time 1781222936.39 total 110.14 last 80.60) - open/close 9027 (time 1781222946.41 total 120.15 last 74.10) - open/close 9778 (time 1781222956.41 total 130.15 last 75.08) total: 10000 open/close in 132.87 seconds: 75.26 ops/second - unlinked 0 (time 1781222964 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 792 (time 1781223073.22 total 10.00 last 79.20) - open/close 1468 (time 1781223083.24 total 20.02 last 67.47) - open/close 2100 (time 1781223093.25 total 30.03 last 63.16) - open/close 2813 (time 1781223103.27 total 40.05 last 71.11) - open/close 3503 (time 1781223113.30 total 50.08 last 68.85) - open/close 4264 (time 1781223123.30 total 60.08 last 76.04) - open/close 5118 (time 1781223133.34 total 70.12 last 85.10) - open/close 5570 (time 1781223143.35 total 80.13 last 45.16) - open/close 6340 (time 1781223153.36 total 90.14 last 76.92) - open/close 7053 (time 1781223163.36 total 100.14 last 71.28) - open/close 7918 (time 1781223173.37 total 110.15 last 86.45) - open/close 8612 (time 1781223183.37 total 120.15 last 69.35) - open/close 9349 (time 1781223193.38 total 130.16 last 73.65) total: 10000 open/close in 139.47 seconds: 71.70 ops/second - unlinked 0 (time 1781223207 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre On OST0, 17190 used inodes rc=0 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.135@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.135@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.201.135@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2