== conf-sanity test 69: replace an OST with the same index ========================================================== 16:44:14 (1787604254) start mds service on oleg336-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 665 (time 1787604366.76 total 10.03 last 66.32) - open/close 1283 (time 1787604376.77 total 20.04 last 61.74) - open/close 1829 (time 1787604386.79 total 30.05 last 54.53) - open/close 2426 (time 1787604396.79 total 40.06 last 59.66) - open/close 3128 (time 1787604406.81 total 50.07 last 70.11) - open/close 3783 (time 1787604416.81 total 60.08 last 65.45) - open/close 4447 (time 1787604426.81 total 70.08 last 66.39) - open/close 5052 (time 1787604436.82 total 80.09 last 60.45) - open/close 5670 (time 1787604446.83 total 90.09 last 61.76) - open/close 6270 (time 1787604456.85 total 100.11 last 59.89) - open/close 6981 (time 1787604466.87 total 110.13 last 70.95) - open/close 7652 (time 1787604476.89 total 120.15 last 66.98) - open/close 8294 (time 1787604486.92 total 130.18 last 63.98) - open/close 8951 (time 1787604496.92 total 140.19 last 65.67) - open/close 9603 (time 1787604506.93 total 150.19 last 65.17) total: 10000 open/close in 155.76 seconds: 64.20 ops/second - unlinked 0 (time 1787604516 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 719 (time 1787604619.20 total 10.01 last 71.86) - open/close 1424 (time 1787604629.21 total 20.02 last 70.43) - open/close 2052 (time 1787604639.21 total 30.02 last 62.80) - open/close 2747 (time 1787604649.22 total 40.03 last 69.44) - open/close 3476 (time 1787604659.22 total 50.03 last 72.87) - open/close 4156 (time 1787604669.22 total 60.03 last 67.99) - open/close 4725 (time 1787604679.23 total 70.04 last 56.86) - open/close 5367 (time 1787604689.24 total 80.04 last 64.17) - open/close 5987 (time 1787604699.25 total 90.05 last 61.93) - open/close 6624 (time 1787604709.26 total 100.07 last 63.60) - open/close 7134 (time 1787604719.29 total 110.09 last 50.88) - open/close 7656 (time 1787604729.33 total 120.14 last 51.96) - open/close 8228 (time 1787604739.35 total 130.15 last 57.11) - open/close 8847 (time 1787604749.35 total 140.16 last 61.86) - open/close 9482 (time 1787604759.36 total 150.17 last 63.45) total: 10000 open/close in 157.92 seconds: 63.32 ops/second - unlinked 0 (time 1787604771 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 661 (time 1787604878.26 total 10.02 last 65.94) - open/close 1320 (time 1787604888.27 total 20.04 last 65.82) - open/close 1979 (time 1787604898.29 total 30.06 last 65.78) - open/close 2536 (time 1787604908.31 total 40.08 last 55.55) - open/close 3135 (time 1787604918.32 total 50.09 last 59.88) - open/close 3760 (time 1787604928.33 total 60.09 last 62.45) - open/close 4247 (time 1787604938.35 total 70.11 last 48.61) - open/close 4950 (time 1787604948.36 total 80.12 last 70.22) - open/close 5651 (time 1787604958.36 total 90.13 last 70.07) - open/close 6336 (time 1787604968.37 total 100.14 last 68.45) - open/close 7051 (time 1787604978.37 total 110.14 last 71.49) - open/close 7698 (time 1787604988.39 total 120.16 last 64.55) - open/close 8415 (time 1787604998.40 total 130.17 last 71.63) - open/close 9087 (time 1787605008.41 total 140.18 last 67.13) - open/close 9852 (time 1787605018.42 total 150.19 last 76.46) total: 10000 open/close in 152.16 seconds: 65.72 ops/second - unlinked 0 (time 1787605025 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 625 (time 1787605136.47 total 10.01 last 62.45) - open/close 1107 (time 1787605146.48 total 20.02 last 48.15) - open/close 1629 (time 1787605156.50 total 30.04 last 52.09) - open/close 2178 (time 1787605166.51 total 40.04 last 54.88) - open/close 2830 (time 1787605176.52 total 50.06 last 65.10) - open/close 3397 (time 1787605186.53 total 60.07 last 56.65) - open/close 4079 (time 1787605196.54 total 70.08 last 68.14) - open/close 4676 (time 1787605206.55 total 80.09 last 59.65) - open/close 5313 (time 1787605216.57 total 90.10 last 63.59) - open/close 5955 (time 1787605226.58 total 100.11 last 64.12) - open/close 6609 (time 1787605236.59 total 110.12 last 65.33) - open/close 7231 (time 1787605246.61 total 120.14 last 62.08) - open/close 7911 (time 1787605256.61 total 130.15 last 67.95) - open/close 8570 (time 1787605266.64 total 140.18 last 65.73) - open/close 9173 (time 1787605276.64 total 150.18 last 60.28) - open/close 9694 (time 1787605286.65 total 160.18 last 52.08) total: 10000 open/close in 165.78 seconds: 60.32 ops/second - unlinked 0 (time 1787605296 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 655 (time 1787605406.68 total 10.01 last 65.41) - open/close 1316 (time 1787605416.70 total 20.03 last 65.98) - open/close 1989 (time 1787605426.71 total 30.04 last 67.23) - open/close 2643 (time 1787605436.72 total 40.04 last 65.39) - open/close 3236 (time 1787605446.72 total 50.05 last 59.29) - open/close 3880 (time 1787605456.72 total 60.05 last 64.37) - open/close 4539 (time 1787605466.73 total 70.06 last 65.84) - open/close 5218 (time 1787605476.74 total 80.07 last 67.83) - open/close 5853 (time 1787605486.75 total 90.08 last 63.46) - open/close 6443 (time 1787605496.75 total 100.08 last 58.97) - open/close 7080 (time 1787605506.78 total 110.11 last 63.52) - open/close 7695 (time 1787605516.79 total 120.12 last 61.46) - open/close 8358 (time 1787605526.80 total 130.12 last 66.25) - open/close 8874 (time 1787605536.81 total 140.13 last 51.55) - open/close 9443 (time 1787605546.81 total 150.13 last 56.90) - open/close 9917 (time 1787605556.81 total 160.14 last 47.38) total: 10000 open/close in 162.08 seconds: 61.70 ops/second - unlinked 0 (time 1787605564 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 698 (time 1787605674.87 total 10.00 last 69.79) - open/close 1379 (time 1787605684.88 total 20.01 last 68.03) - open/close 2110 (time 1787605694.89 total 30.02 last 73.08) - open/close 2832 (time 1787605704.90 total 40.02 last 72.13) - open/close 3444 (time 1787605714.90 total 50.03 last 61.15) - open/close 3975 (time 1787605724.93 total 60.06 last 52.97) - open/close 4596 (time 1787605734.94 total 70.06 last 62.06) - open/close 5172 (time 1787605744.95 total 80.07 last 57.54) - open/close 5744 (time 1787605754.96 total 90.08 last 57.14) - open/close 6174 (time 1787605764.96 total 100.09 last 42.98) - open/close 6649 (time 1787605774.98 total 110.11 last 47.43) - open/close 7191 (time 1787605784.98 total 120.11 last 54.19) - open/close 7855 (time 1787605794.99 total 130.11 last 66.35) - open/close 8506 (time 1787605804.99 total 140.12 last 65.07) - open/close 9145 (time 1787605815.00 total 150.13 last 63.84) - open/close 9744 (time 1787605825.00 total 160.13 last 59.88) total: 10000 open/close in 163.78 seconds: 61.06 ops/second - unlinked 0 (time 1787605833 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 626 (time 1787605946.87 total 10.01 last 62.51) - open/close 1241 (time 1787605956.88 total 20.02 last 61.45) - open/close 1866 (time 1787605966.89 total 30.03 last 62.44) - open/close 2547 (time 1787605976.91 total 40.05 last 67.97) - open/close 3234 (time 1787605986.93 total 50.07 last 68.55) - open/close 3968 (time 1787605996.95 total 60.09 last 73.25) - open/close 4670 (time 1787606006.96 total 70.10 last 70.13) - open/close 5391 (time 1787606016.97 total 80.11 last 72.04) - open/close 6101 (time 1787606026.98 total 90.12 last 70.89) - open/close 6807 (time 1787606036.99 total 100.13 last 70.57) - open/close 7531 (time 1787606047.00 total 110.15 last 72.27) - open/close 8220 (time 1787606057.02 total 120.16 last 68.82) - open/close 8801 (time 1787606067.03 total 130.18 last 58.00) - open/close 9382 (time 1787606077.04 total 140.18 last 58.05) - open/close 9944 (time 1787606087.06 total 150.20 last 56.09) total: 10000 open/close in 151.12 seconds: 66.17 ops/second - unlinked 0 (time 1787606092 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 749 (time 1787606193.12 total 10.01 last 74.81) - open/close 1487 (time 1787606203.13 total 20.02 last 73.76) - open/close 2152 (time 1787606213.13 total 30.02 last 66.46) - open/close 2850 (time 1787606223.13 total 40.03 last 69.78) - open/close 3543 (time 1787606233.15 total 50.04 last 69.19) - open/close 4179 (time 1787606243.16 total 60.05 last 63.53) - open/close 4828 (time 1787606253.17 total 70.06 last 64.84) - open/close 5503 (time 1787606263.19 total 80.08 last 67.39) - open/close 6186 (time 1787606273.19 total 90.08 last 68.29) - open/close 6865 (time 1787606283.19 total 100.08 last 67.87) - open/close 7605 (time 1787606293.20 total 110.10 last 73.91) - open/close 8323 (time 1787606303.21 total 120.10 last 71.80) - open/close 9048 (time 1787606313.22 total 130.11 last 72.43) - open/close 9698 (time 1787606323.23 total 140.12 last 64.91) total: 10000 open/close in 144.08 seconds: 69.41 ops/second - unlinked 0 (time 1787606331 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 669 (time 1787606432.01 total 10.00 last 66.87) - open/close 1335 (time 1787606442.03 total 20.02 last 66.52) - open/close 1961 (time 1787606452.03 total 30.02 last 62.57) - open/close 2620 (time 1787606462.03 total 40.02 last 65.88) - open/close 3292 (time 1787606472.04 total 50.03 last 67.13) - open/close 3919 (time 1787606482.05 total 60.04 last 62.66) - open/close 4566 (time 1787606492.08 total 70.07 last 64.50) - open/close 5208 (time 1787606502.09 total 80.08 last 64.17) - open/close 5844 (time 1787606512.09 total 90.08 last 63.59) - open/close 6499 (time 1787606522.09 total 100.08 last 65.50) - open/close 7177 (time 1787606532.10 total 110.09 last 67.74) - open/close 7808 (time 1787606542.10 total 120.09 last 63.08) - open/close 8432 (time 1787606552.12 total 130.11 last 62.31) - open/close 9108 (time 1787606562.13 total 140.12 last 67.53) - open/close 9712 (time 1787606572.13 total 150.12 last 60.36) total: 10000 open/close in 154.42 seconds: 64.76 ops/second - unlinked 0 (time 1787606581 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 579 (time 1787606686.87 total 10.01 last 57.84) - open/close 1137 (time 1787606696.88 total 20.02 last 55.77) - open/close 1750 (time 1787606706.88 total 30.02 last 61.27) - open/close 2400 (time 1787606716.90 total 40.04 last 64.91) - open/close 2973 (time 1787606726.91 total 50.05 last 57.21) - open/close 3553 (time 1787606736.93 total 60.07 last 57.90) - open/close 4131 (time 1787606746.94 total 70.08 last 57.74) - open/close 4687 (time 1787606756.95 total 80.08 last 55.57) - open/close 5367 (time 1787606766.96 total 90.10 last 67.91) - open/close 6074 (time 1787606776.97 total 100.10 last 70.65) - open/close 6686 (time 1787606787.00 total 110.13 last 61.01) - open/close 7229 (time 1787606797.02 total 120.16 last 54.18) - open/close 7775 (time 1787606807.03 total 130.17 last 54.56) - open/close 8360 (time 1787606817.03 total 140.17 last 58.47) - open/close 9047 (time 1787606827.04 total 150.18 last 68.64) - open/close 9682 (time 1787606837.05 total 160.19 last 63.44) total: 10000 open/close in 165.80 seconds: 60.31 ops/second - unlinked 0 (time 1787606847 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.136@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.136@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.136@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.136@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.136@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.136@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.203.136@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2