== conf-sanity test 69: replace an OST with the same index ========================================================== 01:52:11 (1787118731) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 623 (time 1787118823.87 total 10.00 last 62.27) - open/close 1092 (time 1787118833.89 total 20.03 last 46.80) - open/close 1454 (time 1787118843.93 total 30.06 last 36.08) - open/close 2034 (time 1787118853.94 total 40.07 last 57.95) - open/close 2640 (time 1787118863.94 total 50.07 last 60.58) - open/close 3296 (time 1787118873.94 total 60.07 last 65.60) - open/close 3934 (time 1787118883.95 total 70.08 last 63.76) - open/close 4642 (time 1787118893.96 total 80.09 last 70.73) - open/close 5331 (time 1787118903.97 total 90.10 last 68.82) - open/close 5962 (time 1787118913.97 total 100.10 last 63.09) - open/close 6614 (time 1787118923.98 total 110.11 last 65.13) - open/close 7280 (time 1787118933.99 total 120.12 last 66.52) - open/close 7912 (time 1787118944.01 total 130.14 last 63.09) - open/close 8587 (time 1787118954.02 total 140.15 last 67.43) - open/close 9288 (time 1787118964.02 total 150.15 last 70.08) - open/close 9962 (time 1787118974.02 total 160.16 last 67.38) total: 10000 open/close in 160.68 seconds: 62.24 ops/second - unlinked 0 (time 1787118979 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 630 (time 1787119096.02 total 10.02 last 62.85) - open/close 1208 (time 1787119106.03 total 20.03 last 57.78) - open/close 1828 (time 1787119116.03 total 30.03 last 61.96) - open/close 2441 (time 1787119126.07 total 40.07 last 61.08) - open/close 3081 (time 1787119136.10 total 50.10 last 63.80) - open/close 3744 (time 1787119146.11 total 60.12 last 66.20) - open/close 4381 (time 1787119156.12 total 70.12 last 63.69) - open/close 4991 (time 1787119166.12 total 80.12 last 60.99) - open/close 5547 (time 1787119176.13 total 90.13 last 55.52) - open/close 6152 (time 1787119186.13 total 100.14 last 60.49) - open/close 6739 (time 1787119196.14 total 110.14 last 58.66) - open/close 7381 (time 1787119206.14 total 120.15 last 64.17) - open/close 7997 (time 1787119216.15 total 130.15 last 61.59) - open/close 8699 (time 1787119226.15 total 140.15 last 70.18) - open/close 9211 (time 1787119236.18 total 150.18 last 51.02) - open/close 9800 (time 1787119246.18 total 160.19 last 58.89) total: 10000 open/close in 163.18 seconds: 61.28 ops/second - unlinked 0 (time 1787119254 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 624 (time 1787119372.05 total 10.00 last 62.39) - open/close 1169 (time 1787119382.07 total 20.02 last 54.42) - open/close 1854 (time 1787119392.08 total 30.03 last 68.43) - open/close 2482 (time 1787119402.08 total 40.03 last 62.78) - open/close 3163 (time 1787119412.08 total 50.03 last 68.09) - open/close 3859 (time 1787119422.08 total 60.03 last 69.59) - open/close 4544 (time 1787119432.10 total 70.06 last 68.35) - open/close 5234 (time 1787119442.12 total 80.07 last 68.88) - open/close 5902 (time 1787119452.13 total 90.09 last 66.71) - open/close 6521 (time 1787119462.14 total 100.09 last 61.86) - open/close 7173 (time 1787119472.15 total 110.10 last 65.17) - open/close 7791 (time 1787119482.15 total 120.10 last 61.79) - open/close 8421 (time 1787119492.15 total 130.10 last 62.98) - open/close 9080 (time 1787119502.16 total 140.11 last 65.83) - open/close 9797 (time 1787119512.19 total 150.14 last 71.51) total: 10000 open/close in 153.67 seconds: 65.07 ops/second - unlinked 0 (time 1787119520 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 691 (time 1787119634.94 total 10.01 last 69.05) - open/close 1266 (time 1787119644.94 total 20.01 last 57.49) - open/close 1999 (time 1787119654.94 total 30.02 last 73.25) - open/close 2690 (time 1787119664.95 total 40.02 last 69.06) - open/close 3437 (time 1787119674.96 total 50.03 last 74.64) - open/close 4009 (time 1787119684.96 total 60.03 last 57.19) - open/close 4620 (time 1787119694.97 total 70.04 last 61.06) - open/close 5296 (time 1787119704.99 total 80.06 last 67.47) - open/close 5975 (time 1787119714.99 total 90.07 last 67.85) - open/close 6688 (time 1787119725.00 total 100.07 last 71.27) - open/close 7403 (time 1787119735.01 total 110.08 last 71.42) - open/close 8121 (time 1787119745.02 total 120.09 last 71.74) - open/close 8816 (time 1787119755.03 total 130.10 last 69.42) - open/close 9494 (time 1787119765.04 total 140.11 last 67.71) total: 10000 open/close in 148.48 seconds: 67.35 ops/second - unlinked 0 (time 1787119778 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 677 (time 1787119885.33 total 10.00 last 67.70) - open/close 1290 (time 1787119895.34 total 20.01 last 61.24) - open/close 1947 (time 1787119905.35 total 30.02 last 65.63) - open/close 2588 (time 1787119915.37 total 40.04 last 63.99) - open/close 3197 (time 1787119925.38 total 50.05 last 60.84) - open/close 3814 (time 1787119935.38 total 60.05 last 61.67) - open/close 4474 (time 1787119945.38 total 70.05 last 65.99) - open/close 5163 (time 1787119955.39 total 80.06 last 68.87) - open/close 5848 (time 1787119965.40 total 90.07 last 68.41) - open/close 6544 (time 1787119975.42 total 100.09 last 69.48) - open/close 7242 (time 1787119985.42 total 110.09 last 69.79) - open/close 7917 (time 1787119995.44 total 120.11 last 67.35) - open/close 8538 (time 1787120005.44 total 130.11 last 62.09) - open/close 9229 (time 1787120015.45 total 140.12 last 69.05) - open/close 9867 (time 1787120025.45 total 150.12 last 63.77) total: 10000 open/close in 152.38 seconds: 65.63 ops/second - unlinked 0 (time 1787120032 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 665 (time 1787120148.57 total 10.01 last 66.45) - open/close 1281 (time 1787120158.58 total 20.01 last 61.56) - open/close 1963 (time 1787120168.59 total 30.02 last 68.14) - open/close 2660 (time 1787120178.59 total 40.02 last 69.69) - open/close 3343 (time 1787120188.60 total 50.03 last 68.24) - open/close 4014 (time 1787120198.60 total 60.04 last 67.07) - open/close 4694 (time 1787120208.61 total 70.04 last 67.94) - open/close 5359 (time 1787120218.62 total 80.06 last 66.42) - open/close 6051 (time 1787120228.62 total 90.06 last 69.18) - open/close 6709 (time 1787120238.63 total 100.06 last 65.79) - open/close 7429 (time 1787120248.63 total 110.06 last 71.99) - open/close 8100 (time 1787120258.63 total 120.07 last 67.07) - open/close 8776 (time 1787120268.65 total 130.09 last 67.44) - open/close 9460 (time 1787120278.65 total 140.09 last 68.39) total: 10000 open/close in 147.85 seconds: 67.64 ops/second - unlinked 0 (time 1787120290 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 655 (time 1787120402.23 total 10.01 last 65.46) - open/close 1313 (time 1787120412.24 total 20.02 last 65.74) - open/close 2013 (time 1787120422.24 total 30.02 last 69.98) - open/close 2665 (time 1787120432.25 total 40.02 last 65.17) - open/close 3323 (time 1787120442.26 total 50.04 last 65.71) - open/close 3962 (time 1787120452.26 total 60.04 last 63.90) - open/close 4615 (time 1787120462.27 total 70.04 last 65.27) - open/close 5282 (time 1787120472.27 total 80.05 last 66.67) - open/close 5920 (time 1787120482.28 total 90.06 last 63.71) - open/close 6599 (time 1787120492.30 total 100.08 last 67.77) - open/close 7262 (time 1787120502.33 total 110.11 last 66.11) - open/close 7917 (time 1787120512.34 total 120.11 last 65.47) - open/close 8587 (time 1787120522.35 total 130.13 last 66.88) - open/close 9264 (time 1787120532.38 total 140.15 last 67.54) - open/close 9974 (time 1787120542.39 total 150.17 last 70.92) total: 10000 open/close in 150.49 seconds: 66.45 ops/second - unlinked 0 (time 1787120546 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 566 (time 1787120660.41 total 10.00 last 56.58) - open/close 1144 (time 1787120670.41 total 20.00 last 57.80) - open/close 1804 (time 1787120680.42 total 30.02 last 65.91) - open/close 2438 (time 1787120690.44 total 40.03 last 63.31) - open/close 3031 (time 1787120700.45 total 50.04 last 59.23) - open/close 3670 (time 1787120710.46 total 60.05 last 63.84) - open/close 4255 (time 1787120720.46 total 70.06 last 58.46) - open/close 4869 (time 1787120730.47 total 80.06 last 61.37) - open/close 5594 (time 1787120740.47 total 90.07 last 72.48) - open/close 6207 (time 1787120750.49 total 100.08 last 61.20) - open/close 6884 (time 1787120760.49 total 110.09 last 67.66) - open/close 7518 (time 1787120770.49 total 120.09 last 63.39) - open/close 8143 (time 1787120780.50 total 130.10 last 62.45) - open/close 8751 (time 1787120790.51 total 140.10 last 60.78) - open/close 9421 (time 1787120800.51 total 150.10 last 66.98) total: 10000 open/close in 159.90 seconds: 62.54 ops/second - unlinked 0 (time 1787120814 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 697 (time 1787120923.10 total 10.00 last 69.69) - open/close 1390 (time 1787120933.11 total 20.01 last 69.24) - open/close 2043 (time 1787120943.12 total 30.02 last 65.23) - open/close 2729 (time 1787120953.13 total 40.03 last 68.53) - open/close 3408 (time 1787120963.13 total 50.03 last 67.90) - open/close 4101 (time 1787120973.15 total 60.05 last 69.16) - open/close 4769 (time 1787120983.17 total 70.07 last 66.68) - open/close 5424 (time 1787120993.17 total 80.07 last 65.49) - open/close 6075 (time 1787121003.18 total 90.08 last 65.06) - open/close 6764 (time 1787121013.19 total 100.10 last 68.77) - open/close 7408 (time 1787121023.20 total 110.10 last 64.36) - open/close 8102 (time 1787121033.21 total 120.11 last 69.34) - open/close 8776 (time 1787121043.22 total 130.12 last 67.34) - open/close 9442 (time 1787121053.23 total 140.13 last 66.55) total: 10000 open/close in 149.77 seconds: 66.77 ops/second - unlinked 0 (time 1787121068 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 604 (time 1787121190.82 total 10.00 last 60.39) - open/close 1294 (time 1787121200.82 total 20.00 last 68.99) - open/close 1932 (time 1787121210.82 total 30.00 last 63.79) - open/close 2572 (time 1787121220.82 total 40.01 last 63.99) - open/close 3216 (time 1787121230.83 total 50.02 last 64.33) - open/close 3851 (time 1787121240.84 total 60.02 last 63.45) - open/close 4472 (time 1787121250.87 total 70.05 last 61.96) - open/close 5111 (time 1787121260.87 total 80.05 last 63.88) - open/close 5765 (time 1787121270.88 total 90.06 last 65.34) - open/close 6393 (time 1787121280.89 total 100.07 last 62.75) - open/close 7019 (time 1787121290.89 total 110.07 last 62.57) - open/close 7705 (time 1787121300.90 total 120.08 last 68.53) - open/close 8320 (time 1787121310.91 total 130.09 last 61.46) - open/close 8901 (time 1787121320.92 total 140.10 last 58.05) - open/close 9441 (time 1787121330.93 total 150.11 last 53.93) total: 10000 open/close in 159.65 seconds: 62.64 ops/second - unlinked 0 (time 1787121345 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre On OST0, 15758 used inodes rc=0 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.121@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.121@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.121@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2