== conf-sanity test 69: replace an OST with the same index ========================================================== 09:58:46 (1781272726)
start mds service on oleg233-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg233-server'
oleg233-server: oleg233-server.virtnet: executing load_modules_local
oleg233-server: Loading modules from /home/green/git/lustre-release/lustre
oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg233-server: detected 4 online CPUs by sysfs
oleg233-server: MODOPTS_LIBCFS=
oleg233-server: Force libcfs to create 2 CPU partitions
oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg233-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 560 (time 1781272819.64 total 10.00 last 55.97)
 - open/close 1143 (time 1781272829.65 total 20.01 last 58.27)
 - open/close 1760 (time 1781272839.66 total 30.02 last 61.63)
 - open/close 2386 (time 1781272849.67 total 40.03 last 62.57)
 - open/close 3027 (time 1781272859.67 total 50.03 last 64.08)
 - open/close 3499 (time 1781272869.68 total 60.04 last 47.15)
 - open/close 4181 (time 1781272879.69 total 70.05 last 68.15)
 - open/close 4917 (time 1781272889.69 total 80.05 last 73.60)
 - open/close 5618 (time 1781272899.69 total 90.05 last 70.06)
 - open/close 6466 (time 1781272909.70 total 100.06 last 84.76)
 - open/close 7226 (time 1781272919.70 total 110.06 last 75.99)
 - open/close 7920 (time 1781272929.70 total 120.06 last 69.36)
 - open/close 8552 (time 1781272939.72 total 130.08 last 63.13)
 - open/close 9340 (time 1781272949.72 total 140.08 last 78.79)
total: 10000 open/close in 148.70 seconds: 67.25 ops/second
 - unlinked 0 (time 1781272962 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 629 (time 1781273061.78 total 10.02 last 62.78)
 - open/close 1222 (time 1781273071.80 total 20.04 last 59.18)
 - open/close 1776 (time 1781273081.82 total 30.06 last 55.29)
 - open/close 2532 (time 1781273091.82 total 40.06 last 75.58)
 - open/close 3331 (time 1781273101.83 total 50.07 last 79.81)
 - open/close 4298 (time 1781273111.84 total 60.08 last 96.61)
 - open/close 5090 (time 1781273121.85 total 70.09 last 79.15)
 - open/close 5862 (time 1781273131.85 total 80.09 last 77.20)
 - open/close 6703 (time 1781273141.85 total 90.09 last 84.10)
 - open/close 7482 (time 1781273151.85 total 100.09 last 77.89)
 - open/close 8265 (time 1781273161.85 total 110.09 last 78.29)
 - open/close 8987 (time 1781273171.86 total 120.10 last 72.15)
 - open/close 9774 (time 1781273181.87 total 130.10 last 78.65)
total: 10000 open/close in 132.82 seconds: 75.29 ops/second
 - unlinked 0 (time 1781273190 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 686 (time 1781273286.99 total 10.02 last 68.46)
 - open/close 1366 (time 1781273297.01 total 20.04 last 67.84)
 - open/close 2164 (time 1781273307.01 total 30.05 last 79.78)
 - open/close 2641 (time 1781273317.02 total 40.05 last 47.66)
 - open/close 3168 (time 1781273327.03 total 50.06 last 52.67)
 - open/close 3883 (time 1781273337.05 total 60.08 last 71.35)
 - open/close 4596 (time 1781273347.05 total 70.09 last 71.27)
 - open/close 5364 (time 1781273357.06 total 80.10 last 76.71)
 - open/close 6083 (time 1781273367.07 total 90.11 last 71.84)
 - open/close 6880 (time 1781273377.08 total 100.11 last 79.65)
 - open/close 7625 (time 1781273387.09 total 110.12 last 74.45)
 - open/close 8435 (time 1781273397.09 total 120.12 last 80.98)
 - open/close 9191 (time 1781273407.09 total 130.13 last 75.58)
 - open/close 9962 (time 1781273417.09 total 140.13 last 77.09)
total: 10000 open/close in 140.62 seconds: 71.11 ops/second
 - unlinked 0 (time 1781273422 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 755 (time 1781273523.74 total 10.02 last 75.37)
 - open/close 1576 (time 1781273533.74 total 20.02 last 82.06)
 - open/close 2400 (time 1781273543.75 total 30.03 last 82.37)
 - open/close 3092 (time 1781273553.75 total 40.03 last 69.19)
 - open/close 3851 (time 1781273563.75 total 50.03 last 75.87)
 - open/close 4704 (time 1781273573.76 total 60.04 last 85.25)
 - open/close 5535 (time 1781273583.76 total 70.04 last 83.05)
 - open/close 6258 (time 1781273593.77 total 80.05 last 72.26)
 - open/close 6951 (time 1781273603.78 total 90.06 last 69.24)
 - open/close 7628 (time 1781273613.83 total 100.11 last 67.35)
 - open/close 8344 (time 1781273623.84 total 110.12 last 71.53)
 - open/close 9148 (time 1781273633.85 total 120.13 last 80.34)
 - open/close 9874 (time 1781273643.86 total 130.14 last 72.53)
total: 10000 open/close in 131.48 seconds: 76.05 ops/second
 - unlinked 0 (time 1781273649 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 796 (time 1781273750.77 total 10.00 last 79.56)
 - open/close 1519 (time 1781273760.77 total 20.01 last 72.29)
 - open/close 2102 (time 1781273770.77 total 30.01 last 58.29)
 - open/close 2708 (time 1781273780.78 total 40.01 last 60.59)
 - open/close 3323 (time 1781273790.79 total 50.02 last 61.44)
 - open/close 4087 (time 1781273800.79 total 60.02 last 76.37)
 - open/close 4738 (time 1781273810.79 total 70.03 last 65.07)
 - open/close 5479 (time 1781273820.79 total 80.03 last 74.09)
 - open/close 6173 (time 1781273830.80 total 90.03 last 69.36)
 - open/close 6922 (time 1781273840.81 total 100.04 last 74.82)
 - open/close 7567 (time 1781273850.82 total 110.05 last 64.46)
 - open/close 8188 (time 1781273860.84 total 120.08 last 61.94)
 - open/close 8848 (time 1781273870.86 total 130.09 last 65.91)
 - open/close 9619 (time 1781273880.87 total 140.10 last 77.01)
total: 10000 open/close in 145.15 seconds: 68.89 ops/second
 - unlinked 0 (time 1781273890 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 692 (time 1781274003.45 total 10.01 last 69.16)
 - open/close 1289 (time 1781274013.45 total 20.01 last 59.67)
 - open/close 2058 (time 1781274023.47 total 30.02 last 76.79)
 - open/close 2849 (time 1781274033.47 total 40.02 last 79.09)
 - open/close 3601 (time 1781274043.48 total 50.03 last 75.15)
 - open/close 4381 (time 1781274053.48 total 60.04 last 77.94)
 - open/close 5206 (time 1781274063.49 total 70.05 last 82.44)
 - open/close 5845 (time 1781274073.50 total 80.06 last 63.81)
 - open/close 6716 (time 1781274083.51 total 90.07 last 87.04)
 - open/close 7526 (time 1781274093.52 total 100.08 last 80.92)
 - open/close 8277 (time 1781274103.52 total 110.08 last 75.08)
 - open/close 9000 (time 1781274113.54 total 120.09 last 72.21)
 - open/close 9629 (time 1781274123.55 total 130.11 last 62.80)
total: 10000 open/close in 135.54 seconds: 73.78 ops/second
 - unlinked 0 (time 1781274133 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 761 (time 1781274236.40 total 10.03 last 75.90)
 - open/close 1499 (time 1781274246.40 total 20.03 last 73.77)
 - open/close 2343 (time 1781274256.42 total 30.05 last 84.25)
 - open/close 3170 (time 1781274266.42 total 40.05 last 82.69)
 - open/close 3799 (time 1781274276.43 total 50.06 last 62.83)
 - open/close 4497 (time 1781274286.44 total 60.07 last 69.73)
 - open/close 5275 (time 1781274296.45 total 70.07 last 77.78)
 - open/close 6008 (time 1781274306.45 total 80.08 last 73.27)
 - open/close 6803 (time 1781274316.45 total 90.08 last 79.49)
 - open/close 7535 (time 1781274326.46 total 100.09 last 73.13)
 - open/close 8352 (time 1781274336.46 total 110.09 last 81.69)
 - open/close 9142 (time 1781274346.47 total 120.10 last 78.93)
 - open/close 9951 (time 1781274356.47 total 130.10 last 80.86)
total: 10000 open/close in 130.69 seconds: 76.52 ops/second
 - unlinked 0 (time 1781274361 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 842 (time 1781274456.35 total 10.01 last 84.12)
 - open/close 1650 (time 1781274466.35 total 20.01 last 80.79)
 - open/close 2446 (time 1781274476.35 total 30.01 last 79.59)
 - open/close 3174 (time 1781274486.35 total 40.02 last 72.78)
 - open/close 3912 (time 1781274496.36 total 50.03 last 73.73)
 - open/close 4746 (time 1781274506.39 total 60.05 last 83.23)
 - open/close 5571 (time 1781274516.39 total 70.05 last 82.48)
 - open/close 6348 (time 1781274526.40 total 80.06 last 77.62)
 - open/close 7165 (time 1781274536.41 total 90.07 last 81.59)
 - open/close 7947 (time 1781274546.42 total 100.08 last 78.16)
 - open/close 8811 (time 1781274556.42 total 110.08 last 86.36)
 - open/close 9575 (time 1781274566.43 total 120.09 last 76.33)
total: 10000 open/close in 125.78 seconds: 79.51 ops/second
 - unlinked 0 (time 1781274576 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 844 (time 1781274671.41 total 10.01 last 84.36)
 - open/close 1728 (time 1781274681.42 total 20.01 last 88.34)
 - open/close 2594 (time 1781274691.42 total 30.02 last 86.56)
 - open/close 3421 (time 1781274701.42 total 40.02 last 82.69)
 - open/close 4241 (time 1781274711.42 total 50.02 last 82.00)
 - open/close 5068 (time 1781274721.43 total 60.03 last 82.64)
 - open/close 5835 (time 1781274731.43 total 70.03 last 76.66)
 - open/close 6595 (time 1781274741.44 total 80.04 last 75.94)
 - open/close 7469 (time 1781274751.46 total 90.05 last 87.27)
 - open/close 8368 (time 1781274761.46 total 100.05 last 89.90)
 - open/close 9159 (time 1781274771.46 total 110.06 last 79.06)
total: 10000 open/close in 119.84 seconds: 83.45 ops/second
 - unlinked 0 (time 1781274785 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 1001 (time 1781274880.01 total 10.00 last 100.06)
 - open/close 1804 (time 1781274890.02 total 20.01 last 80.23)
 - open/close 2515 (time 1781274900.02 total 30.02 last 71.07)
 - open/close 3239 (time 1781274910.03 total 40.02 last 72.35)
 - open/close 3985 (time 1781274920.03 total 50.03 last 74.54)
 - open/close 4747 (time 1781274930.05 total 60.04 last 76.12)
 - open/close 5605 (time 1781274940.05 total 70.04 last 85.78)
 - open/close 6396 (time 1781274950.05 total 80.05 last 79.07)
 - open/close 7161 (time 1781274960.06 total 90.06 last 76.43)
 - open/close 8021 (time 1781274970.06 total 100.06 last 86.00)
 - open/close 8916 (time 1781274980.08 total 110.08 last 89.34)
 - open/close 9698 (time 1781274990.08 total 120.08 last 78.18)
total: 10000 open/close in 123.69 seconds: 80.85 ops/second
 - unlinked 0 (time 1781274997 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg233-server: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg233-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.133@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.133@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.133@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.133@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg233-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1
Started lustre-OST0000
oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg233-server: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg233-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg233-client.virtnet:  -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg233-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg233-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server
stop mds service on oleg233-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server

unloading modules via unload_modules_local on: 'oleg233-server'
oleg233-server: oleg233-server.virtnet: executing unload_modules_local
oleg233-server: 
modules unloaded.
pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2
pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2