== conf-sanity test 69: replace an OST with the same index ========================================================== 15:55:47 (1784404547)
start mds service on oleg102-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg102-server'
oleg102-server: oleg102-server.virtnet: executing load_modules_local
oleg102-server: Loading modules from /home/green/git/lustre-release/lustre
oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg102-server: detected 4 online CPUs by sysfs
oleg102-server: MODOPTS_LIBCFS=
oleg102-server: Force libcfs to create 2 CPU partitions
oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg102-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg102-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1
Started lustre-OST0000
oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg102-client.virtnet:  -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 767 (time 1784404617.95 total 10.01 last 76.60)
 - open/close 1557 (time 1784404627.97 total 20.03 last 78.89)
 - open/close 2225 (time 1784404637.97 total 30.03 last 66.79)
 - open/close 2964 (time 1784404647.97 total 40.03 last 73.90)
 - open/close 3659 (time 1784404657.98 total 50.04 last 69.40)
 - open/close 4371 (time 1784404667.99 total 60.05 last 71.16)
 - open/close 5127 (time 1784404677.99 total 70.05 last 75.57)
 - open/close 5899 (time 1784404688.00 total 80.06 last 77.13)
 - open/close 6611 (time 1784404698.01 total 90.07 last 71.15)
 - open/close 7337 (time 1784404708.02 total 100.08 last 72.52)
 - open/close 8119 (time 1784404718.02 total 110.08 last 78.18)
 - open/close 8849 (time 1784404728.03 total 120.09 last 72.96)
 - open/close 9614 (time 1784404738.03 total 130.09 last 76.49)
total: 10000 open/close in 134.78 seconds: 74.20 ops/second
 - unlinked 0 (time 1784404746 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 864 (time 1784404843.44 total 10.01 last 86.27)
 - open/close 1630 (time 1784404853.46 total 20.03 last 76.47)
 - open/close 2352 (time 1784404863.47 total 30.04 last 72.16)
 - open/close 3196 (time 1784404873.47 total 40.04 last 84.37)
 - open/close 3971 (time 1784404883.47 total 50.05 last 77.46)
 - open/close 4805 (time 1784404893.49 total 60.06 last 83.24)
 - open/close 5601 (time 1784404903.50 total 70.07 last 79.57)
 - open/close 6368 (time 1784404913.54 total 80.11 last 76.41)
 - open/close 7131 (time 1784404923.55 total 90.12 last 76.20)
 - open/close 8065 (time 1784404933.57 total 100.14 last 93.23)
 - open/close 9033 (time 1784404943.57 total 110.14 last 96.78)
 - open/close 9780 (time 1784404953.57 total 120.14 last 74.69)
total: 10000 open/close in 122.89 seconds: 81.38 ops/second
 - unlinked 0 (time 1784404959 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 833 (time 1784405045.28 total 10.00 last 83.30)
 - open/close 1697 (time 1784405055.28 total 20.00 last 86.39)
 - open/close 2596 (time 1784405065.28 total 30.01 last 89.86)
 - open/close 3394 (time 1784405075.29 total 40.01 last 79.75)
 - open/close 4235 (time 1784405085.30 total 50.03 last 83.98)
 - open/close 5142 (time 1784405095.32 total 60.04 last 90.55)
 - open/close 6086 (time 1784405105.33 total 70.05 last 94.29)
 - open/close 6994 (time 1784405115.33 total 80.06 last 90.79)
 - open/close 8025 (time 1784405125.34 total 90.06 last 103.06)
 - open/close 9034 (time 1784405135.34 total 100.06 last 100.87)
total: 10000 open/close in 109.83 seconds: 91.05 ops/second
 - unlinked 0 (time 1784405147 ; total 0 ; last 0)
total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second
 - open/close 1130 (time 1784405223.63 total 10.00 last 112.99)
 - open/close 2215 (time 1784405233.63 total 20.00 last 108.48)
 - open/close 3118 (time 1784405243.64 total 30.01 last 90.22)
 - open/close 4094 (time 1784405253.64 total 40.01 last 97.60)
 - open/close 5021 (time 1784405263.67 total 50.04 last 92.48)
 - open/close 6018 (time 1784405273.67 total 60.04 last 99.63)
 - open/close 7053 (time 1784405283.68 total 70.05 last 103.44)
 - open/close 8012 (time 1784405293.69 total 80.05 last 95.85)
 - open/close 9034 (time 1784405303.69 total 90.06 last 102.10)
 - open/close 9964 (time 1784405313.71 total 100.08 last 92.87)
total: 10000 open/close in 100.53 seconds: 99.48 ops/second
 - unlinked 0 (time 1784405316 ; total 0 ; last 0)
total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second
 - open/close 962 (time 1784405385.43 total 10.01 last 96.12)
 - open/close 2071 (time 1784405395.44 total 20.01 last 110.87)
 - open/close 3229 (time 1784405405.44 total 30.01 last 115.78)
 - open/close 4462 (time 1784405415.44 total 40.02 last 123.24)
 - open/close 5460 (time 1784405425.45 total 50.03 last 99.69)
 - open/close 6459 (time 1784405435.46 total 60.03 last 99.86)
 - open/close 7541 (time 1784405445.47 total 70.04 last 108.11)
 - open/close 8804 (time 1784405455.47 total 80.05 last 126.23)
 - open/close 9916 (time 1784405465.48 total 90.06 last 111.09)
total: 10000 open/close in 90.79 seconds: 110.15 ops/second
 - unlinked 0 (time 1784405468 ; total 0 ; last 0)
total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second
 - open/close 1055 (time 1784405530.50 total 10.00 last 105.50)
 - open/close 2293 (time 1784405540.50 total 20.00 last 123.79)
 - open/close 3989 (time 1784405550.50 total 30.00 last 169.54)
 - open/close 6161 (time 1784405560.51 total 40.01 last 217.11)
 - open/close 7550 (time 1784405570.51 total 50.01 last 138.87)
 - open/close 8926 (time 1784405580.52 total 60.02 last 137.51)
total: 10000 open/close in 68.06 seconds: 146.92 ops/second
 - unlinked 0 (time 1784405590 ; total 0 ; last 0)
total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second
 - open/close 1212 (time 1784405652.19 total 10.00 last 121.20)
 - open/close 2566 (time 1784405662.20 total 20.01 last 135.26)
 - open/close 3693 (time 1784405672.20 total 30.01 last 112.69)
 - open/close 4741 (time 1784405682.21 total 40.02 last 104.76)
 - open/close 5808 (time 1784405692.22 total 50.02 last 106.61)
 - open/close 7001 (time 1784405702.22 total 60.03 last 119.26)
 - open/close 8159 (time 1784405712.22 total 70.03 last 115.73)
 - open/close 9236 (time 1784405722.23 total 80.04 last 107.62)
total: 10000 open/close in 86.05 seconds: 116.21 ops/second
 - unlinked 0 (time 1784405730 ; total 0 ; last 0)
total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second
 - open/close 1091 (time 1784405790.96 total 10.01 last 109.02)
 - open/close 2366 (time 1784405800.97 total 20.01 last 127.42)
 - open/close 3513 (time 1784405810.97 total 30.01 last 114.68)
 - open/close 4939 (time 1784405820.98 total 40.02 last 142.49)
 - open/close 6188 (time 1784405830.98 total 50.03 last 124.84)
 - open/close 7716 (time 1784405840.98 total 60.03 last 152.79)
 - open/close 9171 (time 1784405850.99 total 70.03 last 145.41)
total: 10000 open/close in 74.47 seconds: 134.28 ops/second
 - unlinked 0 (time 1784405857 ; total 0 ; last 0)
total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second
 - open/close 1097 (time 1784405907.33 total 10.00 last 109.65)
 - open/close 2144 (time 1784405917.34 total 20.01 last 104.59)
 - open/close 3210 (time 1784405927.34 total 30.02 last 106.57)
 - open/close 4293 (time 1784405937.34 total 40.02 last 108.27)
 - open/close 5431 (time 1784405947.35 total 50.03 last 113.72)
 - open/close 6556 (time 1784405957.35 total 60.03 last 112.49)
 - open/close 7674 (time 1784405967.36 total 70.04 last 111.70)
 - open/close 8861 (time 1784405977.36 total 80.04 last 118.69)
 - open/close 9992 (time 1784405987.37 total 90.05 last 112.98)
total: 10000 open/close in 90.12 seconds: 110.96 ops/second
 - unlinked 0 (time 1784405989 ; total 0 ; last 0)
total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second
 - open/close 1415 (time 1784406044.71 total 10.00 last 141.44)
 - open/close 2902 (time 1784406054.72 total 20.01 last 148.57)
 - open/close 4195 (time 1784406064.73 total 30.02 last 129.24)
 - open/close 5598 (time 1784406074.73 total 40.02 last 140.23)
 - open/close 7534 (time 1784406084.73 total 50.02 last 193.58)
 - open/close 9413 (time 1784406094.73 total 60.03 last 187.87)
total: 10000 open/close in 63.71 seconds: 156.97 ops/second
 - unlinked 0 (time 1784406100 ; total 0 ; last 0)
total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg102-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg102-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg102-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1
Started lustre-OST0000
oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg102-client.virtnet:  -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre
On OST0, 19414 used inodes rc=0
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg102-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg102-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.102@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.102@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.102@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg102-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1
Started lustre-OST0000
oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg102-client.virtnet:  -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg102-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg102-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server
stop mds service on oleg102-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server

unloading modules via unload_modules_local on: 'oleg102-server'
oleg102-server: oleg102-server.virtnet: executing unload_modules_local
oleg102-server: 
modules unloaded.
pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2
pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2
pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2