== conf-sanity test 69: replace an OST with the same index ========================================================== 08:13:22 (1786191202)
start mds service on oleg160-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: MODOPTS_LIBCFS=
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 799 (time 1786191285.68 total 10.00 last 79.89)
 - open/close 1543 (time 1786191295.69 total 20.01 last 74.35)
 - open/close 2287 (time 1786191305.69 total 30.01 last 74.37)
 - open/close 2928 (time 1786191315.72 total 40.03 last 63.96)
 - open/close 3637 (time 1786191325.72 total 50.04 last 70.88)
 - open/close 4343 (time 1786191335.72 total 60.04 last 70.56)
 - open/close 5054 (time 1786191345.73 total 70.05 last 71.08)
 - open/close 5902 (time 1786191355.73 total 80.05 last 84.75)
 - open/close 6679 (time 1786191365.73 total 90.05 last 77.69)
 - open/close 7496 (time 1786191375.74 total 100.06 last 81.65)
 - open/close 8309 (time 1786191385.76 total 110.08 last 81.12)
 - open/close 9074 (time 1786191395.78 total 120.10 last 76.36)
 - open/close 9838 (time 1786191405.78 total 130.10 last 76.39)
total: 10000 open/close in 132.05 seconds: 75.73 ops/second
 - unlinked 0 (time 1786191411 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 796 (time 1786191507.64 total 10.01 last 79.51)
 - open/close 1684 (time 1786191517.65 total 20.02 last 88.72)
 - open/close 2500 (time 1786191527.66 total 30.03 last 81.50)
 - open/close 3492 (time 1786191537.67 total 40.04 last 99.13)
 - open/close 4328 (time 1786191547.68 total 50.05 last 83.50)
 - open/close 5063 (time 1786191557.68 total 60.05 last 73.50)
 - open/close 5872 (time 1786191567.69 total 70.06 last 80.81)
 - open/close 6705 (time 1786191577.71 total 80.08 last 83.13)
 - open/close 7522 (time 1786191587.72 total 90.09 last 81.62)
 - open/close 8330 (time 1786191597.73 total 100.10 last 80.76)
 - open/close 9183 (time 1786191607.74 total 110.11 last 85.16)
total: 10000 open/close in 119.49 seconds: 83.69 ops/second
 - unlinked 0 (time 1786191620 ; total 0 ; last 0)
total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second
 - open/close 880 (time 1786191697.63 total 10.00 last 87.98)
 - open/close 1793 (time 1786191707.64 total 20.01 last 91.21)
 - open/close 2634 (time 1786191717.65 total 30.02 last 84.01)
 - open/close 3462 (time 1786191727.65 total 40.02 last 82.78)
 - open/close 4257 (time 1786191737.65 total 50.03 last 79.49)
 - open/close 5162 (time 1786191747.66 total 60.03 last 90.47)
 - open/close 6074 (time 1786191757.67 total 70.04 last 91.11)
 - open/close 6998 (time 1786191767.67 total 80.04 last 92.36)
 - open/close 7873 (time 1786191777.68 total 90.05 last 87.45)
 - open/close 8723 (time 1786191787.69 total 100.06 last 84.91)
 - open/close 9788 (time 1786191797.69 total 110.07 last 106.41)
total: 10000 open/close in 112.57 seconds: 88.83 ops/second
 - unlinked 0 (time 1786191803 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 1036 (time 1786191885.92 total 10.00 last 103.59)
 - open/close 2118 (time 1786191895.93 total 20.02 last 108.04)
 - open/close 3041 (time 1786191905.94 total 30.02 last 92.23)
 - open/close 4074 (time 1786191915.94 total 40.02 last 103.28)
 - open/close 5075 (time 1786191925.95 total 50.03 last 100.06)
 - open/close 6202 (time 1786191935.96 total 60.04 last 112.59)
 - open/close 7233 (time 1786191945.98 total 70.06 last 102.92)
 - open/close 8241 (time 1786191955.98 total 80.06 last 100.75)
 - open/close 9250 (time 1786191965.99 total 90.07 last 100.83)
total: 10000 open/close in 96.79 seconds: 103.31 ops/second
 - unlinked 0 (time 1786191975 ; total 0 ; last 0)
total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second
 - open/close 1023 (time 1786192044.00 total 10.01 last 102.22)
 - open/close 2087 (time 1786192054.00 total 20.01 last 106.38)
 - open/close 3091 (time 1786192064.00 total 30.01 last 100.38)
 - open/close 4057 (time 1786192074.01 total 40.02 last 96.52)
 - open/close 5037 (time 1786192084.01 total 50.02 last 98.00)
 - open/close 6193 (time 1786192094.02 total 60.03 last 115.49)
 - open/close 7340 (time 1786192104.02 total 70.03 last 114.66)
 - open/close 8400 (time 1786192114.04 total 80.05 last 105.83)
 - open/close 9320 (time 1786192124.04 total 90.06 last 91.93)
total: 10000 open/close in 96.69 seconds: 103.42 ops/second
 - unlinked 0 (time 1786192133 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 1090 (time 1786192205.63 total 10.00 last 109.00)
 - open/close 2148 (time 1786192215.64 total 20.01 last 105.68)
 - open/close 3254 (time 1786192225.65 total 30.02 last 110.55)
 - open/close 4384 (time 1786192235.66 total 40.02 last 112.91)
 - open/close 5474 (time 1786192245.66 total 50.03 last 108.97)
 - open/close 6578 (time 1786192255.66 total 60.03 last 110.40)
 - open/close 7689 (time 1786192265.67 total 70.03 last 111.03)
 - open/close 8775 (time 1786192275.67 total 80.04 last 108.58)
 - open/close 9858 (time 1786192285.67 total 90.04 last 108.23)
total: 10000 open/close in 91.22 seconds: 109.62 ops/second
 - unlinked 0 (time 1786192289 ; total 0 ; last 0)
total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second
 - open/close 1239 (time 1786192349.67 total 10.00 last 123.86)
 - open/close 2422 (time 1786192359.68 total 20.01 last 118.24)
 - open/close 3584 (time 1786192369.69 total 30.02 last 116.08)
 - open/close 4814 (time 1786192379.69 total 40.03 last 122.91)
 - open/close 5996 (time 1786192389.70 total 50.03 last 118.14)
 - open/close 7274 (time 1786192399.70 total 60.03 last 127.78)
 - open/close 8636 (time 1786192409.71 total 70.04 last 136.06)
 - open/close 9844 (time 1786192419.71 total 80.04 last 120.78)
total: 10000 open/close in 81.26 seconds: 123.06 ops/second
 - unlinked 0 (time 1786192423 ; total 0 ; last 0)
total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second
 - open/close 1293 (time 1786192477.11 total 10.00 last 129.24)
 - open/close 2515 (time 1786192487.11 total 20.01 last 122.19)
 - open/close 3671 (time 1786192497.12 total 30.01 last 115.55)
 - open/close 4897 (time 1786192507.12 total 40.01 last 122.55)
 - open/close 6114 (time 1786192517.13 total 50.02 last 121.63)
 - open/close 7398 (time 1786192527.13 total 60.03 last 128.32)
 - open/close 8604 (time 1786192537.13 total 70.03 last 120.57)
total: 10000 open/close in 78.61 seconds: 127.21 ops/second
 - unlinked 0 (time 1786192547 ; total 0 ; last 0)
total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second
 - open/close 1348 (time 1786192602.29 total 10.01 last 134.72)
 - open/close 2625 (time 1786192612.30 total 20.02 last 127.55)
 - open/close 4066 (time 1786192622.30 total 30.02 last 144.06)
 - open/close 5415 (time 1786192632.31 total 40.02 last 134.85)
 - open/close 6966 (time 1786192642.31 total 50.02 last 155.08)
 - open/close 8430 (time 1786192652.31 total 60.02 last 146.40)
 - open/close 9862 (time 1786192662.31 total 70.03 last 143.12)
total: 10000 open/close in 70.73 seconds: 141.39 ops/second
 - unlinked 0 (time 1786192664 ; total 0 ; last 0)
total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second
 - open/close 1609 (time 1786192709.38 total 10.01 last 160.81)
 - open/close 3074 (time 1786192719.38 total 20.01 last 146.49)
 - open/close 4462 (time 1786192729.38 total 30.01 last 138.78)
 - open/close 5838 (time 1786192739.39 total 40.02 last 137.47)
 - open/close 7176 (time 1786192749.39 total 50.02 last 133.73)
 - open/close 8648 (time 1786192759.40 total 60.03 last 147.16)
 - open/close 9963 (time 1786192769.40 total 70.03 last 131.48)
total: 10000 open/close in 70.33 seconds: 142.18 ops/second
 - unlinked 0 (time 1786192771 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
On OST0, 11390 used inodes rc=0
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.160@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.160@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server

unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: 
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2