== conf-sanity test 69: replace an OST with the same index ========================================================== 19:56:52 (1787788612)
start mds service on oleg139-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg139-server'
oleg139-server: oleg139-server.virtnet: executing load_modules_local
oleg139-server: Loading modules from /home/green/git/lustre-release/lustre
oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg139-server: detected 4 online CPUs by sysfs
oleg139-server: MODOPTS_LIBCFS=
oleg139-server: Force libcfs to create 2 CPU partitions
oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 641 (time 1787788707.42 total 10.05 last 63.75)
 - open/close 1334 (time 1787788717.43 total 20.06 last 69.25)
 - open/close 1955 (time 1787788727.44 total 30.07 last 62.03)
 - open/close 2706 (time 1787788737.45 total 40.09 last 74.99)
 - open/close 3488 (time 1787788747.46 total 50.09 last 78.15)
 - open/close 4089 (time 1787788757.48 total 60.11 last 60.02)
 - open/close 4714 (time 1787788767.48 total 70.11 last 62.50)
 - open/close 5448 (time 1787788777.48 total 80.11 last 73.36)
 - open/close 6289 (time 1787788787.48 total 90.11 last 84.09)
 - open/close 6973 (time 1787788797.49 total 100.12 last 68.37)
 - open/close 7697 (time 1787788807.50 total 110.13 last 72.29)
 - open/close 8342 (time 1787788817.52 total 120.15 last 64.37)
 - open/close 9070 (time 1787788827.53 total 130.16 last 72.77)
 - open/close 9726 (time 1787788837.53 total 140.16 last 65.57)
total: 10000 open/close in 145.39 seconds: 68.78 ops/second
 - unlinked 0 (time 1787788848 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 652 (time 1787788955.67 total 10.01 last 65.11)
 - open/close 1366 (time 1787788965.68 total 20.02 last 71.38)
 - open/close 2128 (time 1787788975.70 total 30.04 last 76.04)
 - open/close 2802 (time 1787788985.71 total 40.05 last 67.33)
 - open/close 3432 (time 1787788995.72 total 50.06 last 62.95)
 - open/close 4113 (time 1787789005.72 total 60.07 last 68.04)
 - open/close 4784 (time 1787789015.73 total 70.08 last 67.03)
 - open/close 5517 (time 1787789025.74 total 80.08 last 73.24)
 - open/close 6254 (time 1787789035.75 total 90.09 last 73.67)
 - open/close 7002 (time 1787789045.75 total 100.09 last 74.77)
 - open/close 7693 (time 1787789055.77 total 110.11 last 68.99)
 - open/close 8366 (time 1787789065.77 total 120.11 last 67.25)
 - open/close 9072 (time 1787789075.79 total 130.13 last 70.52)
 - open/close 9820 (time 1787789085.80 total 140.14 last 74.67)
total: 10000 open/close in 142.96 seconds: 69.95 ops/second
 - unlinked 0 (time 1787789093 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 670 (time 1787789194.26 total 10.00 last 66.97)
 - open/close 1381 (time 1787789204.26 total 20.01 last 71.09)
 - open/close 2059 (time 1787789214.27 total 30.01 last 67.77)
 - open/close 2790 (time 1787789224.27 total 40.01 last 73.08)
 - open/close 3456 (time 1787789234.28 total 50.02 last 66.55)
 - open/close 4117 (time 1787789244.29 total 60.03 last 66.04)
 - open/close 4640 (time 1787789254.31 total 70.06 last 52.15)
 - open/close 5229 (time 1787789264.32 total 80.06 last 58.88)
 - open/close 5878 (time 1787789274.33 total 90.07 last 64.84)
 - open/close 6526 (time 1787789284.34 total 100.08 last 64.74)
 - open/close 7232 (time 1787789294.35 total 110.09 last 70.52)
 - open/close 7902 (time 1787789304.35 total 120.09 last 67.00)
 - open/close 8768 (time 1787789314.36 total 130.10 last 86.52)
 - open/close 9534 (time 1787789324.36 total 140.11 last 76.55)
total: 10000 open/close in 146.42 seconds: 68.30 ops/second
 - unlinked 0 (time 1787789335 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 732 (time 1787789436.83 total 10.00 last 73.17)
 - open/close 1412 (time 1787789446.87 total 20.04 last 67.73)
 - open/close 2066 (time 1787789456.88 total 30.05 last 65.34)
 - open/close 2699 (time 1787789466.90 total 40.07 last 63.20)
 - open/close 3212 (time 1787789476.91 total 50.08 last 51.24)
 - open/close 3663 (time 1787789486.92 total 60.09 last 45.06)
 - open/close 4240 (time 1787789496.93 total 70.10 last 57.64)
 - open/close 4652 (time 1787789506.93 total 80.10 last 41.18)
 - open/close 5272 (time 1787789516.94 total 90.12 last 61.92)
 - open/close 5955 (time 1787789526.95 total 100.12 last 68.28)
 - open/close 6724 (time 1787789536.97 total 110.14 last 76.73)
 - open/close 7614 (time 1787789546.97 total 120.14 last 88.99)
 - open/close 8332 (time 1787789556.98 total 130.16 last 71.71)
 - open/close 9052 (time 1787789566.99 total 140.16 last 71.95)
 - open/close 9825 (time 1787789577.01 total 150.18 last 77.18)
total: 10000 open/close in 152.51 seconds: 65.57 ops/second
 - unlinked 0 (time 1787789583 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 704 (time 1787789679.12 total 10.01 last 70.32)
 - open/close 1419 (time 1787789689.13 total 20.02 last 71.45)
 - open/close 2219 (time 1787789699.13 total 30.02 last 79.98)
 - open/close 2956 (time 1787789709.14 total 40.03 last 73.62)
 - open/close 3742 (time 1787789719.15 total 50.04 last 78.52)
 - open/close 4430 (time 1787789729.17 total 60.06 last 68.66)
 - open/close 5145 (time 1787789739.17 total 70.07 last 71.48)
 - open/close 5878 (time 1787789749.18 total 80.08 last 73.22)
 - open/close 6597 (time 1787789759.21 total 90.11 last 71.68)
 - open/close 7396 (time 1787789769.22 total 100.11 last 79.84)
 - open/close 8207 (time 1787789779.23 total 110.12 last 81.04)
 - open/close 8989 (time 1787789789.24 total 120.13 last 78.10)
 - open/close 9663 (time 1787789799.25 total 130.14 last 67.35)
total: 10000 open/close in 134.99 seconds: 74.08 ops/second
 - unlinked 0 (time 1787789808 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 728 (time 1787789902.45 total 10.01 last 72.76)
 - open/close 1481 (time 1787789912.45 total 20.01 last 75.29)
 - open/close 2207 (time 1787789922.46 total 30.01 last 72.57)
 - open/close 2877 (time 1787789932.46 total 40.02 last 66.96)
 - open/close 3678 (time 1787789942.47 total 50.02 last 80.03)
 - open/close 4416 (time 1787789952.48 total 60.03 last 73.73)
 - open/close 5177 (time 1787789962.49 total 70.04 last 76.04)
 - open/close 5980 (time 1787789972.50 total 80.05 last 80.25)
 - open/close 6702 (time 1787789982.51 total 90.06 last 72.10)
 - open/close 7489 (time 1787789992.53 total 100.08 last 78.57)
 - open/close 8411 (time 1787790002.53 total 110.08 last 92.16)
 - open/close 9180 (time 1787790012.53 total 120.09 last 76.86)
 - open/close 9818 (time 1787790022.54 total 130.09 last 63.75)
total: 10000 open/close in 132.33 seconds: 75.57 ops/second
 - unlinked 0 (time 1787790029 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 689 (time 1787790129.10 total 10.00 last 68.89)
 - open/close 1408 (time 1787790139.10 total 20.01 last 71.87)
 - open/close 2196 (time 1787790149.11 total 30.01 last 78.78)
 - open/close 2983 (time 1787790159.12 total 40.02 last 78.63)
 - open/close 3754 (time 1787790169.12 total 50.03 last 77.03)
 - open/close 4558 (time 1787790179.13 total 60.03 last 80.37)
 - open/close 5224 (time 1787790189.13 total 70.03 last 66.59)
 - open/close 5932 (time 1787790199.14 total 80.05 last 70.70)
 - open/close 6658 (time 1787790209.15 total 90.05 last 72.57)
 - open/close 7365 (time 1787790219.15 total 100.05 last 70.70)
 - open/close 8176 (time 1787790229.16 total 110.06 last 81.04)
 - open/close 9016 (time 1787790239.19 total 120.09 last 83.75)
 - open/close 9821 (time 1787790249.19 total 130.09 last 80.47)
total: 10000 open/close in 132.27 seconds: 75.60 ops/second
 - unlinked 0 (time 1787790254 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 878 (time 1787790346.79 total 10.01 last 87.70)
 - open/close 1701 (time 1787790356.79 total 20.01 last 82.27)
 - open/close 2572 (time 1787790366.79 total 30.02 last 87.09)
 - open/close 3303 (time 1787790376.81 total 40.03 last 73.01)
 - open/close 4080 (time 1787790386.81 total 50.03 last 77.66)
 - open/close 4865 (time 1787790396.82 total 60.04 last 78.44)
 - open/close 5766 (time 1787790406.82 total 70.04 last 90.09)
 - open/close 6590 (time 1787790416.82 total 80.04 last 82.38)
 - open/close 7436 (time 1787790426.83 total 90.05 last 84.55)
 - open/close 8269 (time 1787790436.83 total 100.06 last 83.27)
 - open/close 9090 (time 1787790446.84 total 110.06 last 82.05)
 - open/close 9906 (time 1787790456.85 total 120.07 last 81.54)
total: 10000 open/close in 121.14 seconds: 82.55 ops/second
 - unlinked 0 (time 1787790461 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 809 (time 1787790555.89 total 10.00 last 80.89)
 - open/close 1601 (time 1787790565.89 total 20.01 last 79.17)
 - open/close 2393 (time 1787790575.90 total 30.01 last 79.16)
 - open/close 3159 (time 1787790585.90 total 40.02 last 76.54)
 - open/close 3934 (time 1787790595.91 total 50.03 last 77.44)
 - open/close 4695 (time 1787790605.92 total 60.03 last 76.06)
 - open/close 5429 (time 1787790615.93 total 70.04 last 73.32)
 - open/close 6193 (time 1787790625.93 total 80.04 last 76.40)
 - open/close 6954 (time 1787790635.93 total 90.05 last 76.07)
 - open/close 7712 (time 1787790645.94 total 100.05 last 75.78)
 - open/close 8415 (time 1787790655.94 total 110.06 last 70.26)
 - open/close 9072 (time 1787790665.94 total 120.06 last 65.68)
 - open/close 9723 (time 1787790675.99 total 130.10 last 64.81)
total: 10000 open/close in 133.93 seconds: 74.67 ops/second
 - unlinked 0 (time 1787790684 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 756 (time 1787790776.04 total 10.01 last 75.55)
 - open/close 1568 (time 1787790786.05 total 20.01 last 81.18)
 - open/close 2383 (time 1787790796.05 total 30.01 last 81.47)
 - open/close 3258 (time 1787790806.06 total 40.02 last 87.44)
 - open/close 4044 (time 1787790816.06 total 50.02 last 78.59)
 - open/close 4722 (time 1787790826.07 total 60.03 last 67.74)
 - open/close 5549 (time 1787790836.07 total 70.03 last 82.68)
 - open/close 6360 (time 1787790846.08 total 80.04 last 81.05)
 - open/close 7128 (time 1787790856.08 total 90.04 last 76.76)
 - open/close 7990 (time 1787790866.09 total 100.05 last 86.14)
 - open/close 8723 (time 1787790876.09 total 110.05 last 73.29)
 - open/close 9471 (time 1787790886.10 total 120.06 last 74.74)
total: 10000 open/close in 128.27 seconds: 77.96 ops/second
 - unlinked 0 (time 1787790898 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
On OST0, 19862 used inodes rc=0
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server
stop mds service on oleg139-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server

unloading modules via unload_modules_local on: 'oleg139-server'
oleg139-server: oleg139-server.virtnet: executing unload_modules_local
oleg139-server: 
modules unloaded.
pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2