== conf-sanity test 69: replace an OST with the same index ========================================================== 09:07:13 (1788440833)
start mds service on oleg159-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg159-server'
oleg159-server: oleg159-server.virtnet: executing load_modules_local
oleg159-server: Loading modules from /home/green/git/lustre-release/lustre
oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg159-server: detected 4 online CPUs by sysfs
oleg159-server: MODOPTS_LIBCFS=
oleg159-server: Force libcfs to create 2 CPU partitions
oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg159-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg159-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Started lustre-OST0000
oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg159-client.virtnet:  -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 722 (time 1788440916.79 total 10.01 last 72.15)
 - open/close 1246 (time 1788440926.80 total 20.02 last 52.34)
 - open/close 1727 (time 1788440936.80 total 30.02 last 48.10)
 - open/close 2362 (time 1788440946.81 total 40.03 last 63.44)
 - open/close 3033 (time 1788440956.82 total 50.04 last 67.02)
 - open/close 3677 (time 1788440966.83 total 60.05 last 64.32)
 - open/close 4318 (time 1788440976.84 total 70.06 last 64.05)
 - open/close 4893 (time 1788440986.84 total 80.07 last 57.47)
 - open/close 5600 (time 1788440996.85 total 90.07 last 70.69)
 - open/close 6305 (time 1788441006.85 total 100.07 last 70.48)
 - open/close 6911 (time 1788441016.86 total 110.08 last 60.56)
 - open/close 7569 (time 1788441026.86 total 120.08 last 65.79)
 - open/close 8327 (time 1788441036.87 total 130.09 last 75.73)
 - open/close 8978 (time 1788441046.87 total 140.09 last 65.09)
 - open/close 9620 (time 1788441056.88 total 150.10 last 64.13)
total: 10000 open/close in 155.57 seconds: 64.28 ops/second
 - unlinked 0 (time 1788441067 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 646 (time 1788441174.31 total 10.00 last 64.58)
 - open/close 1358 (time 1788441184.33 total 20.03 last 71.00)
 - open/close 2009 (time 1788441194.33 total 30.03 last 65.10)
 - open/close 2663 (time 1788441204.34 total 40.03 last 65.39)
 - open/close 3290 (time 1788441214.35 total 50.04 last 62.63)
 - open/close 3972 (time 1788441224.36 total 60.05 last 68.13)
 - open/close 4661 (time 1788441234.36 total 70.06 last 68.87)
 - open/close 5352 (time 1788441244.37 total 80.07 last 69.05)
 - open/close 6122 (time 1788441254.39 total 90.09 last 76.85)
 - open/close 6980 (time 1788441264.41 total 100.11 last 85.63)
 - open/close 7818 (time 1788441274.41 total 110.11 last 83.79)
 - open/close 8548 (time 1788441284.42 total 120.11 last 72.96)
 - open/close 9415 (time 1788441294.42 total 130.12 last 86.64)
total: 10000 open/close in 137.07 seconds: 72.95 ops/second
 - unlinked 0 (time 1788441305 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 822 (time 1788441397.47 total 10.01 last 82.08)
 - open/close 1600 (time 1788441407.47 total 20.02 last 77.74)
 - open/close 2449 (time 1788441417.48 total 30.03 last 84.82)
 - open/close 3397 (time 1788441427.49 total 40.04 last 94.69)
 - open/close 4246 (time 1788441437.51 total 50.06 last 84.79)
 - open/close 5044 (time 1788441447.52 total 60.07 last 79.70)
 - open/close 5883 (time 1788441457.55 total 70.09 last 83.68)
 - open/close 6571 (time 1788441467.56 total 80.10 last 68.73)
 - open/close 7202 (time 1788441477.58 total 90.13 last 62.94)
 - open/close 7989 (time 1788441487.58 total 100.13 last 78.68)
 - open/close 8695 (time 1788441497.59 total 110.14 last 70.55)
 - open/close 9420 (time 1788441507.59 total 120.14 last 72.49)
total: 10000 open/close in 127.08 seconds: 78.69 ops/second
 - unlinked 0 (time 1788441519 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 760 (time 1788441613.34 total 10.01 last 75.94)
 - open/close 1539 (time 1788441623.36 total 20.02 last 77.79)
 - open/close 2266 (time 1788441633.38 total 30.05 last 72.53)
 - open/close 3052 (time 1788441643.39 total 40.05 last 78.55)
 - open/close 3781 (time 1788441653.41 total 50.07 last 72.74)
 - open/close 4636 (time 1788441663.41 total 60.07 last 85.49)
 - open/close 5427 (time 1788441673.41 total 70.08 last 79.09)
 - open/close 6324 (time 1788441683.42 total 80.09 last 89.58)
 - open/close 7182 (time 1788441693.43 total 90.10 last 85.73)
 - open/close 7916 (time 1788441703.45 total 100.12 last 73.27)
 - open/close 8731 (time 1788441713.46 total 110.12 last 81.42)
 - open/close 9390 (time 1788441723.47 total 120.13 last 65.85)
total: 10000 open/close in 127.01 seconds: 78.73 ops/second
 - unlinked 0 (time 1788441735 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 669 (time 1788441834.32 total 10.01 last 66.81)
 - open/close 1429 (time 1788441844.34 total 20.03 last 75.90)
 - open/close 2242 (time 1788441854.34 total 30.03 last 81.29)
 - open/close 3172 (time 1788441864.35 total 40.04 last 92.92)
 - open/close 4148 (time 1788441874.35 total 50.04 last 97.59)
 - open/close 4988 (time 1788441884.35 total 60.04 last 83.97)
 - open/close 5851 (time 1788441894.36 total 70.05 last 86.25)
 - open/close 6587 (time 1788441904.37 total 80.06 last 73.53)
 - open/close 7362 (time 1788441914.37 total 90.06 last 77.46)
 - open/close 8224 (time 1788441924.38 total 100.07 last 86.15)
 - open/close 8965 (time 1788441934.39 total 110.08 last 74.02)
 - open/close 9719 (time 1788441944.40 total 120.09 last 75.32)
total: 10000 open/close in 124.06 seconds: 80.61 ops/second
 - unlinked 0 (time 1788441952 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 785 (time 1788442045.03 total 10.00 last 78.46)
 - open/close 1674 (time 1788442055.04 total 20.01 last 88.84)
 - open/close 2501 (time 1788442065.04 total 30.01 last 82.68)
 - open/close 3335 (time 1788442075.06 total 40.04 last 83.20)
 - open/close 3891 (time 1788442085.07 total 50.05 last 55.55)
 - open/close 4629 (time 1788442095.08 total 60.06 last 73.72)
 - open/close 5293 (time 1788442105.09 total 70.06 last 66.37)
 - open/close 6036 (time 1788442115.10 total 80.07 last 74.23)
 - open/close 6785 (time 1788442125.10 total 90.07 last 74.89)
 - open/close 7247 (time 1788442135.11 total 100.09 last 46.14)
 - open/close 7984 (time 1788442145.12 total 110.10 last 73.61)
 - open/close 8769 (time 1788442155.16 total 120.13 last 78.22)
 - open/close 9432 (time 1788442165.16 total 130.13 last 66.29)
total: 10000 open/close in 136.51 seconds: 73.26 ops/second
 - unlinked 0 (time 1788442175 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 886 (time 1788442268.95 total 10.00 last 88.59)
 - open/close 1731 (time 1788442278.95 total 20.00 last 84.48)
 - open/close 2593 (time 1788442288.95 total 30.00 last 86.19)
 - open/close 3491 (time 1788442298.96 total 40.01 last 89.74)
 - open/close 4338 (time 1788442308.96 total 50.02 last 84.67)
 - open/close 5135 (time 1788442318.97 total 60.02 last 79.63)
 - open/close 6012 (time 1788442328.98 total 70.03 last 87.63)
 - open/close 6886 (time 1788442339.00 total 80.05 last 87.25)
 - open/close 7772 (time 1788442349.00 total 90.05 last 88.58)
 - open/close 8645 (time 1788442359.00 total 100.06 last 87.27)
 - open/close 9605 (time 1788442369.01 total 110.06 last 95.94)
total: 10000 open/close in 114.78 seconds: 87.12 ops/second
 - unlinked 0 (time 1788442377 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 839 (time 1788442475.48 total 10.00 last 83.86)
 - open/close 1724 (time 1788442485.48 total 20.00 last 88.50)
 - open/close 2554 (time 1788442495.48 total 30.01 last 82.99)
 - open/close 3488 (time 1788442505.49 total 40.01 last 93.31)
 - open/close 4353 (time 1788442515.49 total 50.02 last 86.49)
 - open/close 5132 (time 1788442525.50 total 60.03 last 77.82)
 - open/close 6066 (time 1788442535.51 total 70.04 last 93.31)
 - open/close 6917 (time 1788442545.51 total 80.04 last 85.10)
 - open/close 7663 (time 1788442555.53 total 90.06 last 74.42)
 - open/close 8284 (time 1788442565.53 total 100.06 last 62.09)
 - open/close 9131 (time 1788442575.54 total 110.07 last 84.63)
 - open/close 9916 (time 1788442585.55 total 120.08 last 78.44)
total: 10000 open/close in 120.90 seconds: 82.71 ops/second
 - unlinked 0 (time 1788442589 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 728 (time 1788442682.67 total 10.01 last 72.76)
 - open/close 1223 (time 1788442692.67 total 20.01 last 49.50)
 - open/close 1957 (time 1788442702.67 total 30.01 last 73.38)
 - open/close 2779 (time 1788442712.67 total 40.01 last 82.17)
 - open/close 3578 (time 1788442722.68 total 50.01 last 79.87)
 - open/close 4376 (time 1788442732.68 total 60.02 last 79.78)
 - open/close 5290 (time 1788442742.68 total 70.02 last 91.38)
 - open/close 6145 (time 1788442752.69 total 80.02 last 85.47)
 - open/close 7006 (time 1788442762.69 total 90.02 last 86.08)
 - open/close 7848 (time 1788442772.69 total 100.03 last 84.16)
 - open/close 8555 (time 1788442782.70 total 110.03 last 70.66)
 - open/close 9094 (time 1788442792.73 total 120.06 last 53.74)
 - open/close 9664 (time 1788442802.73 total 130.07 last 56.97)
total: 10000 open/close in 136.95 seconds: 73.02 ops/second
 - unlinked 0 (time 1788442814 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 879 (time 1788442908.41 total 10.00 last 87.88)
 - open/close 1687 (time 1788442918.43 total 20.02 last 80.67)
 - open/close 2511 (time 1788442928.44 total 30.03 last 82.35)
 - open/close 3318 (time 1788442938.45 total 40.04 last 80.62)
 - open/close 4146 (time 1788442948.45 total 50.04 last 82.73)
 - open/close 4910 (time 1788442958.47 total 60.06 last 76.29)
 - open/close 5801 (time 1788442968.47 total 70.06 last 89.05)
 - open/close 6589 (time 1788442978.48 total 80.07 last 78.72)
 - open/close 7327 (time 1788442988.49 total 90.08 last 73.77)
 - open/close 7911 (time 1788442998.49 total 100.08 last 58.39)
 - open/close 8548 (time 1788443008.50 total 110.09 last 63.60)
 - open/close 9157 (time 1788443018.53 total 120.12 last 60.72)
 - open/close 9912 (time 1788443028.54 total 130.13 last 75.49)
total: 10000 open/close in 131.25 seconds: 76.19 ops/second
 - unlinked 0 (time 1788443034 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg159-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg159-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg159-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Started lustre-OST0000
oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg159-client.virtnet:  -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg159-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg159-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.159@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.159@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.159@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg159-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Started lustre-OST0000
oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg159-client.virtnet:  -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg159-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg159-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server
stop mds service on oleg159-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server

unloading modules via unload_modules_local on: 'oleg159-server'
oleg159-server: oleg159-server.virtnet: executing unload_modules_local
oleg159-server: 
modules unloaded.
pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2
pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2