== conf-sanity test 69: replace an OST with the same index ========================================================== 17:23:17 (1773609797) start mds service on oleg421-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 708 (time 1773609885.22 total 10.02 last 70.63) - open/close 1352 (time 1773609895.22 total 20.03 last 64.38) - open/close 2042 (time 1773609905.23 total 30.04 last 68.95) - open/close 2684 (time 1773609915.23 total 40.04 last 64.18) - open/close 3411 (time 1773609925.23 total 50.04 last 72.68) - open/close 4089 (time 1773609935.24 total 60.04 last 67.79) - open/close 4856 (time 1773609945.24 total 70.05 last 76.66) - open/close 5474 (time 1773609955.25 total 80.06 last 61.75) - open/close 6154 (time 1773609965.28 total 90.09 last 67.78) - open/close 6788 (time 1773609975.28 total 100.09 last 63.39) - open/close 7531 (time 1773609985.30 total 110.11 last 74.19) - open/close 8254 (time 1773609995.31 total 120.12 last 72.22) - open/close 8937 (time 1773610005.31 total 130.12 last 68.29) - open/close 9665 (time 1773610015.31 total 140.12 last 72.79) total: 10000 open/close in 145.43 seconds: 68.76 ops/second - unlinked 0 (time 1773610025 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 734 (time 1773610141.31 total 10.05 last 73.03) - open/close 1416 (time 1773610151.32 total 20.06 last 68.17) - open/close 2056 (time 1773610161.33 total 30.07 last 63.93) - open/close 2727 (time 1773610171.35 total 40.09 last 66.98) - open/close 3349 (time 1773610181.36 total 50.10 last 62.14) - open/close 4019 (time 1773610191.37 total 60.11 last 66.93) - open/close 4602 (time 1773610201.38 total 70.11 last 58.25) - open/close 5245 (time 1773610211.38 total 80.12 last 64.28) - open/close 5867 (time 1773610221.40 total 90.14 last 62.07) - open/close 6480 (time 1773610231.42 total 100.16 last 61.20) - open/close 7040 (time 1773610241.42 total 110.16 last 55.97) - open/close 7530 (time 1773610251.43 total 120.17 last 48.96) - open/close 8206 (time 1773610261.44 total 130.18 last 67.51) - open/close 8938 (time 1773610271.46 total 140.20 last 73.06) - open/close 9613 (time 1773610281.47 total 150.20 last 67.47) total: 10000 open/close in 155.27 seconds: 64.40 ops/second - unlinked 0 (time 1773610291 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 749 (time 1773610418.30 total 10.01 last 74.84) - open/close 1480 (time 1773610428.31 total 20.02 last 73.03) - open/close 2269 (time 1773610438.33 total 30.03 last 78.77) - open/close 3040 (time 1773610448.34 total 40.04 last 77.03) - open/close 3694 (time 1773610458.34 total 50.05 last 65.38) - open/close 4461 (time 1773610468.35 total 60.06 last 76.61) - open/close 5187 (time 1773610478.36 total 70.06 last 72.55) - open/close 5888 (time 1773610488.36 total 80.07 last 70.08) - open/close 6650 (time 1773610498.36 total 90.07 last 76.19) - open/close 7381 (time 1773610508.37 total 100.08 last 73.04) - open/close 8088 (time 1773610518.37 total 110.08 last 70.69) - open/close 8729 (time 1773610528.39 total 120.09 last 64.00) - open/close 9271 (time 1773610538.39 total 130.09 last 54.19) - open/close 9830 (time 1773610548.40 total 140.11 last 55.83) total: 10000 open/close in 141.69 seconds: 70.58 ops/second - unlinked 0 (time 1773610554 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 681 (time 1773610656.59 total 10.00 last 68.09) - open/close 1396 (time 1773610666.59 total 20.01 last 71.44) - open/close 2031 (time 1773610676.60 total 30.02 last 63.44) - open/close 2617 (time 1773610686.61 total 40.02 last 58.57) - open/close 3273 (time 1773610696.62 total 50.04 last 65.50) - open/close 3902 (time 1773610706.63 total 60.05 last 62.87) - open/close 4430 (time 1773610716.63 total 70.05 last 52.79) - open/close 5117 (time 1773610726.65 total 80.06 last 68.58) - open/close 5703 (time 1773610736.65 total 90.07 last 58.57) - open/close 6383 (time 1773610746.66 total 100.07 last 67.97) - open/close 7034 (time 1773610756.66 total 110.08 last 65.05) - open/close 7701 (time 1773610766.67 total 120.09 last 66.65) - open/close 8368 (time 1773610776.69 total 130.11 last 66.56) - open/close 9046 (time 1773610786.70 total 140.12 last 67.74) - open/close 9614 (time 1773610796.78 total 150.20 last 56.33) total: 10000 open/close in 157.02 seconds: 63.69 ops/second - unlinked 0 (time 1773610810 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 719 (time 1773610919.21 total 10.01 last 71.84) - open/close 1461 (time 1773610929.22 total 20.02 last 74.13) - open/close 2146 (time 1773610939.23 total 30.02 last 68.47) - open/close 2709 (time 1773610949.24 total 40.03 last 56.26) - open/close 3344 (time 1773610959.25 total 50.04 last 63.43) - open/close 3944 (time 1773610969.29 total 60.09 last 59.73) - open/close 4536 (time 1773610979.30 total 70.09 last 59.17) - open/close 5262 (time 1773610989.30 total 80.10 last 72.57) - open/close 6004 (time 1773610999.31 total 90.10 last 74.17) - open/close 6698 (time 1773611009.31 total 100.11 last 69.37) - open/close 7399 (time 1773611019.33 total 110.12 last 69.98) - open/close 7988 (time 1773611029.34 total 120.13 last 58.85) - open/close 8593 (time 1773611039.34 total 130.13 last 60.50) - open/close 9224 (time 1773611049.35 total 140.14 last 63.05) - open/close 9804 (time 1773611059.35 total 150.14 last 58.00) total: 10000 open/close in 153.19 seconds: 65.28 ops/second - unlinked 0 (time 1773611067 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 636 (time 1773611184.31 total 10.01 last 63.53) - open/close 1206 (time 1773611194.32 total 20.01 last 56.98) - open/close 1677 (time 1773611204.32 total 30.02 last 47.08) - open/close 2219 (time 1773611214.33 total 40.02 last 54.18) - open/close 2878 (time 1773611224.33 total 50.03 last 65.88) - open/close 3529 (time 1773611234.34 total 60.04 last 65.03) - open/close 4181 (time 1773611244.36 total 70.06 last 65.08) - open/close 4878 (time 1773611254.38 total 80.08 last 69.54) - open/close 5531 (time 1773611264.38 total 90.08 last 65.28) - open/close 6213 (time 1773611274.39 total 100.08 last 68.19) - open/close 6845 (time 1773611284.39 total 110.09 last 63.17) - open/close 7563 (time 1773611294.41 total 120.10 last 71.68) - open/close 8156 (time 1773611304.42 total 130.11 last 59.25) - open/close 8603 (time 1773611314.42 total 140.12 last 44.67) - open/close 9090 (time 1773611324.43 total 150.13 last 48.67) - open/close 9630 (time 1773611334.43 total 160.13 last 53.99) total: 10000 open/close in 165.30 seconds: 60.49 ops/second - unlinked 0 (time 1773611344 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 732 (time 1773611445.32 total 10.00 last 73.20) - open/close 1340 (time 1773611455.32 total 20.01 last 60.76) - open/close 1993 (time 1773611465.33 total 30.01 last 65.27) - open/close 2657 (time 1773611475.35 total 40.03 last 66.25) - open/close 3283 (time 1773611485.36 total 50.04 last 62.56) - open/close 3715 (time 1773611495.40 total 60.08 last 43.01) - open/close 4298 (time 1773611505.40 total 70.09 last 58.28) - open/close 4833 (time 1773611515.42 total 80.11 last 53.38) - open/close 5437 (time 1773611525.44 total 90.12 last 60.34) - open/close 6075 (time 1773611535.47 total 100.15 last 63.61) - open/close 6757 (time 1773611545.48 total 110.16 last 68.11) - open/close 7252 (time 1773611555.49 total 120.18 last 49.42) - open/close 7753 (time 1773611565.50 total 130.18 last 50.07) - open/close 8344 (time 1773611575.51 total 140.19 last 59.06) - open/close 8929 (time 1773611585.52 total 150.20 last 58.45) - open/close 9676 (time 1773611595.53 total 160.21 last 74.63) total: 10000 open/close in 165.84 seconds: 60.30 ops/second - unlinked 0 (time 1773611606 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 744 (time 1773611710.70 total 10.00 last 74.37) - open/close 1514 (time 1773611720.70 total 20.01 last 76.98) - open/close 2253 (time 1773611730.71 total 30.02 last 73.83) - open/close 2749 (time 1773611740.72 total 40.02 last 49.57) - open/close 3549 (time 1773611750.72 total 50.03 last 79.97) - open/close 4330 (time 1773611760.73 total 60.04 last 78.04) - open/close 4980 (time 1773611770.74 total 70.04 last 64.95) - open/close 5695 (time 1773611780.74 total 80.04 last 71.49) - open/close 6277 (time 1773611790.75 total 90.05 last 58.17) - open/close 6985 (time 1773611800.75 total 100.06 last 70.74) - open/close 7782 (time 1773611810.76 total 110.06 last 79.64) - open/close 8599 (time 1773611820.77 total 120.07 last 81.64) - open/close 9476 (time 1773611830.78 total 130.08 last 87.60) total: 10000 open/close in 136.82 seconds: 73.09 ops/second - unlinked 0 (time 1773611841 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 776 (time 1773611941.19 total 10.01 last 77.54) - open/close 1478 (time 1773611951.21 total 20.02 last 70.12) - open/close 2167 (time 1773611961.21 total 30.02 last 68.88) - open/close 2996 (time 1773611971.21 total 40.03 last 82.87) - open/close 3732 (time 1773611981.22 total 50.03 last 73.57) - open/close 4503 (time 1773611991.23 total 60.04 last 77.03) - open/close 5158 (time 1773612001.24 total 70.05 last 65.43) - open/close 5860 (time 1773612011.24 total 80.06 last 70.16) - open/close 6489 (time 1773612021.25 total 90.07 last 62.83) - open/close 7092 (time 1773612031.26 total 100.07 last 60.27) - open/close 7763 (time 1773612041.26 total 110.07 last 67.08) - open/close 8525 (time 1773612051.27 total 120.09 last 76.11) - open/close 9278 (time 1773612061.28 total 130.09 last 75.24) - open/close 9975 (time 1773612071.28 total 140.10 last 69.68) total: 10000 open/close in 140.47 seconds: 71.19 ops/second - unlinked 0 (time 1773612075 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 660 (time 1773612180.30 total 10.01 last 65.96) - open/close 1253 (time 1773612190.32 total 20.02 last 59.20) - open/close 1894 (time 1773612200.32 total 30.03 last 64.06) - open/close 2473 (time 1773612210.34 total 40.05 last 57.81) - open/close 3114 (time 1773612220.35 total 50.05 last 64.05) - open/close 3781 (time 1773612230.36 total 60.07 last 66.62) - open/close 4455 (time 1773612240.37 total 70.08 last 67.30) - open/close 4984 (time 1773612250.39 total 80.10 last 52.81) - open/close 5657 (time 1773612260.39 total 90.10 last 67.27) - open/close 6346 (time 1773612270.40 total 100.11 last 68.83) - open/close 7125 (time 1773612280.41 total 110.11 last 77.89) - open/close 7854 (time 1773612290.42 total 120.12 last 72.83) - open/close 8572 (time 1773612300.42 total 130.13 last 71.76) - open/close 9262 (time 1773612310.43 total 140.14 last 68.93) - open/close 9982 (time 1773612320.44 total 150.15 last 71.91) total: 10000 open/close in 150.46 seconds: 66.46 ops/second - unlinked 0 (time 1773612326 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.121@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.121@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.204.121@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2