== conf-sanity test 69: replace an OST with the same index ========================================================== 07:55:53 (1787658953)
start mds service on oleg411-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg411-server'
oleg411-server: oleg411-server.virtnet: executing load_modules_local
oleg411-server: Loading modules from /home/green/git/lustre-release/lustre
oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg411-server: detected 4 online CPUs by sysfs
oleg411-server: MODOPTS_LIBCFS=
oleg411-server: Force libcfs to create 2 CPU partitions
oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 675 (time 1787659042.96 total 10.01 last 67.44)
 - open/close 1418 (time 1787659052.96 total 20.01 last 74.27)
 - open/close 2118 (time 1787659062.97 total 30.02 last 69.94)
 - open/close 2905 (time 1787659072.98 total 40.03 last 78.65)
 - open/close 3677 (time 1787659082.98 total 50.03 last 77.16)
 - open/close 4392 (time 1787659092.99 total 60.04 last 71.46)
 - open/close 5059 (time 1787659103.00 total 70.05 last 66.62)
 - open/close 5813 (time 1787659113.01 total 80.06 last 75.34)
 - open/close 6572 (time 1787659123.01 total 90.06 last 75.87)
 - open/close 7221 (time 1787659133.03 total 100.08 last 64.82)
 - open/close 7876 (time 1787659143.03 total 110.08 last 65.44)
 - open/close 8508 (time 1787659153.04 total 120.09 last 63.15)
 - open/close 9248 (time 1787659163.05 total 130.10 last 73.93)
 - open/close 9864 (time 1787659173.06 total 140.11 last 61.55)
total: 10000 open/close in 141.82 seconds: 70.51 ops/second
 - unlinked 0 (time 1787659179 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 681 (time 1787659277.62 total 10.01 last 68.04)
 - open/close 1391 (time 1787659287.63 total 20.02 last 70.91)
 - open/close 2108 (time 1787659297.64 total 30.03 last 71.64)
 - open/close 2846 (time 1787659307.66 total 40.05 last 73.65)
 - open/close 3652 (time 1787659317.68 total 50.07 last 80.47)
 - open/close 4321 (time 1787659327.68 total 60.07 last 66.89)
 - open/close 4985 (time 1787659337.69 total 70.07 last 66.35)
 - open/close 5655 (time 1787659347.70 total 80.09 last 66.89)
 - open/close 6392 (time 1787659357.71 total 90.10 last 73.63)
 - open/close 7120 (time 1787659367.72 total 100.11 last 72.72)
 - open/close 7847 (time 1787659377.73 total 110.12 last 72.61)
 - open/close 8597 (time 1787659387.74 total 120.13 last 74.96)
 - open/close 9347 (time 1787659397.75 total 130.14 last 74.92)
total: 10000 open/close in 138.74 seconds: 72.08 ops/second
 - unlinked 0 (time 1787659411 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 760 (time 1787659512.54 total 10.02 last 75.87)
 - open/close 1534 (time 1787659522.54 total 20.02 last 77.39)
 - open/close 2286 (time 1787659532.55 total 30.02 last 75.16)
 - open/close 2961 (time 1787659542.55 total 40.03 last 67.47)
 - open/close 3701 (time 1787659552.56 total 50.04 last 73.93)
 - open/close 4477 (time 1787659562.56 total 60.04 last 77.56)
 - open/close 5261 (time 1787659572.58 total 70.06 last 78.27)
 - open/close 5948 (time 1787659582.59 total 80.07 last 68.63)
 - open/close 6658 (time 1787659592.59 total 90.07 last 70.98)
 - open/close 7267 (time 1787659602.59 total 100.07 last 60.90)
 - open/close 7894 (time 1787659612.61 total 110.09 last 62.61)
 - open/close 8560 (time 1787659622.61 total 120.09 last 66.55)
 - open/close 9298 (time 1787659632.62 total 130.10 last 73.78)
total: 10000 open/close in 139.46 seconds: 71.71 ops/second
 - unlinked 0 (time 1787659646 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 696 (time 1787659747.93 total 10.01 last 69.55)
 - open/close 1460 (time 1787659757.93 total 20.01 last 76.37)
 - open/close 2212 (time 1787659767.94 total 30.02 last 75.13)
 - open/close 2862 (time 1787659777.95 total 40.03 last 64.96)
 - open/close 3493 (time 1787659787.95 total 50.04 last 63.06)
 - open/close 4223 (time 1787659797.97 total 60.05 last 72.86)
 - open/close 4935 (time 1787659807.98 total 70.07 last 71.12)
 - open/close 5693 (time 1787659817.98 total 80.07 last 75.80)
 - open/close 6480 (time 1787659827.98 total 90.07 last 78.70)
 - open/close 7099 (time 1787659838.00 total 100.09 last 61.77)
 - open/close 7895 (time 1787659848.01 total 110.09 last 79.59)
 - open/close 8669 (time 1787659858.01 total 120.09 last 77.37)
 - open/close 9382 (time 1787659868.01 total 130.09 last 71.29)
total: 10000 open/close in 138.37 seconds: 72.27 ops/second
 - unlinked 0 (time 1787659880 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 731 (time 1787659986.91 total 10.00 last 73.08)
 - open/close 1414 (time 1787659996.93 total 20.02 last 68.17)
 - open/close 2142 (time 1787660006.94 total 30.04 last 72.70)
 - open/close 2849 (time 1787660016.96 total 40.05 last 70.61)
 - open/close 3391 (time 1787660026.96 total 50.05 last 54.19)
 - open/close 4084 (time 1787660036.97 total 60.06 last 69.23)
 - open/close 4820 (time 1787660046.97 total 70.06 last 73.59)
 - open/close 5511 (time 1787660056.97 total 80.07 last 69.07)
 - open/close 6246 (time 1787660066.98 total 90.07 last 73.49)
 - open/close 7056 (time 1787660076.98 total 100.08 last 80.93)
 - open/close 7843 (time 1787660086.99 total 110.08 last 78.67)
 - open/close 8572 (time 1787660096.99 total 120.09 last 72.86)
 - open/close 9276 (time 1787660107.02 total 130.11 last 70.22)
 - open/close 9975 (time 1787660117.02 total 140.12 last 69.86)
total: 10000 open/close in 140.37 seconds: 71.24 ops/second
 - unlinked 0 (time 1787660121 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 728 (time 1787660227.10 total 10.01 last 72.74)
 - open/close 1421 (time 1787660237.12 total 20.03 last 69.12)
 - open/close 2256 (time 1787660247.14 total 30.06 last 83.31)
 - open/close 3037 (time 1787660257.15 total 40.06 last 78.06)
 - open/close 3755 (time 1787660267.15 total 50.06 last 71.78)
 - open/close 4491 (time 1787660277.16 total 60.07 last 73.57)
 - open/close 5195 (time 1787660287.16 total 70.07 last 70.38)
 - open/close 5918 (time 1787660297.16 total 80.08 last 72.26)
 - open/close 6728 (time 1787660307.18 total 90.09 last 80.89)
 - open/close 7498 (time 1787660317.19 total 100.11 last 76.88)
 - open/close 8228 (time 1787660327.21 total 110.12 last 72.87)
 - open/close 8969 (time 1787660337.22 total 120.13 last 74.04)
 - open/close 9701 (time 1787660347.24 total 130.15 last 73.07)
total: 10000 open/close in 134.46 seconds: 74.37 ops/second
 - unlinked 0 (time 1787660355 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 765 (time 1787660456.87 total 10.00 last 76.50)
 - open/close 1528 (time 1787660466.87 total 20.00 last 76.27)
 - open/close 2240 (time 1787660476.87 total 30.01 last 71.19)
 - open/close 2976 (time 1787660486.89 total 40.02 last 73.51)
 - open/close 3762 (time 1787660496.90 total 50.03 last 78.51)
 - open/close 4475 (time 1787660506.93 total 60.07 last 71.04)
 - open/close 5156 (time 1787660516.95 total 70.08 last 68.01)
 - open/close 5819 (time 1787660526.95 total 80.08 last 66.28)
 - open/close 6536 (time 1787660536.95 total 90.09 last 71.68)
 - open/close 7169 (time 1787660546.96 total 100.09 last 63.26)
 - open/close 7852 (time 1787660556.96 total 110.10 last 68.28)
 - open/close 8606 (time 1787660566.98 total 120.11 last 75.29)
 - open/close 9378 (time 1787660576.99 total 130.12 last 77.13)
total: 10000 open/close in 138.57 seconds: 72.17 ops/second
 - unlinked 0 (time 1787660589 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 662 (time 1787660694.15 total 10.00 last 66.20)
 - open/close 1423 (time 1787660704.16 total 20.01 last 76.06)
 - open/close 2106 (time 1787660714.17 total 30.01 last 68.25)
 - open/close 2838 (time 1787660724.17 total 40.01 last 73.19)
 - open/close 3563 (time 1787660734.18 total 50.02 last 72.44)
 - open/close 4173 (time 1787660744.19 total 60.03 last 60.93)
 - open/close 4842 (time 1787660754.20 total 70.04 last 66.85)
 - open/close 5550 (time 1787660764.20 total 80.04 last 70.80)
 - open/close 6225 (time 1787660774.20 total 90.05 last 67.48)
 - open/close 6902 (time 1787660784.20 total 100.05 last 67.67)
 - open/close 7634 (time 1787660794.22 total 110.07 last 73.08)
 - open/close 8380 (time 1787660804.24 total 120.08 last 74.49)
 - open/close 9094 (time 1787660814.26 total 130.10 last 71.24)
 - open/close 9741 (time 1787660824.27 total 140.11 last 64.65)
total: 10000 open/close in 143.51 seconds: 69.68 ops/second
 - unlinked 0 (time 1787660832 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 704 (time 1787660931.91 total 10.01 last 70.36)
 - open/close 1393 (time 1787660941.92 total 20.01 last 68.85)
 - open/close 2127 (time 1787660951.93 total 30.02 last 73.32)
 - open/close 2865 (time 1787660961.94 total 40.03 last 73.75)
 - open/close 3597 (time 1787660971.96 total 50.05 last 73.05)
 - open/close 4317 (time 1787660981.97 total 60.06 last 71.94)
 - open/close 5002 (time 1787660991.97 total 70.06 last 68.49)
 - open/close 5698 (time 1787661001.98 total 80.07 last 69.54)
 - open/close 6498 (time 1787661011.98 total 90.07 last 79.98)
 - open/close 7231 (time 1787661022.00 total 100.09 last 73.19)
 - open/close 7960 (time 1787661032.01 total 110.11 last 72.77)
 - open/close 8651 (time 1787661042.03 total 120.12 last 69.00)
 - open/close 9299 (time 1787661052.03 total 130.13 last 64.77)
 - open/close 9967 (time 1787661062.05 total 140.14 last 66.71)
total: 10000 open/close in 140.66 seconds: 71.09 ops/second
 - unlinked 0 (time 1787661067 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 743 (time 1787661170.16 total 10.01 last 74.24)
 - open/close 1459 (time 1787661180.18 total 20.04 last 71.40)
 - open/close 2134 (time 1787661190.20 total 30.06 last 67.37)
 - open/close 2834 (time 1787661200.21 total 40.06 last 69.99)
 - open/close 3635 (time 1787661210.22 total 50.07 last 80.02)
 - open/close 4400 (time 1787661220.22 total 60.07 last 76.49)
 - open/close 5109 (time 1787661230.23 total 70.08 last 70.79)
 - open/close 5874 (time 1787661240.24 total 80.09 last 76.45)
 - open/close 6599 (time 1787661250.24 total 90.09 last 72.49)
 - open/close 7291 (time 1787661260.25 total 100.10 last 69.16)
 - open/close 7947 (time 1787661270.26 total 110.11 last 65.50)
 - open/close 8622 (time 1787661280.27 total 120.12 last 67.45)
 - open/close 9331 (time 1787661290.29 total 130.14 last 70.79)
 - open/close 9937 (time 1787661300.29 total 140.14 last 60.57)
total: 10000 open/close in 141.12 seconds: 70.86 ops/second
 - unlinked 0 (time 1787661306 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.111@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.111@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.111@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server
stop mds service on oleg411-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server

unloading modules via unload_modules_local on: 'oleg411-server'
oleg411-server: oleg411-server.virtnet: executing unload_modules_local
oleg411-server: 
modules unloaded.
pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2