== conf-sanity test 69: replace an OST with the same index ========================================================== 15:28:33 (1778614113)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg241-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 599 (time 1778614208.54 total 10.00 last 59.87)
 - open/close 1267 (time 1778614218.56 total 20.03 last 66.65)
 - open/close 1948 (time 1778614228.57 total 30.04 last 68.04)
 - open/close 2610 (time 1778614238.59 total 40.05 last 66.09)
 - open/close 3136 (time 1778614248.60 total 50.06 last 52.55)
 - open/close 3818 (time 1778614258.61 total 60.07 last 68.12)
 - open/close 4630 (time 1778614268.61 total 70.08 last 81.17)
 - open/close 5317 (time 1778614278.62 total 80.08 last 68.68)
 - open/close 5946 (time 1778614288.64 total 90.10 last 62.79)
 - open/close 6755 (time 1778614298.64 total 100.11 last 80.84)
 - open/close 7485 (time 1778614308.65 total 110.11 last 72.98)
 - open/close 8215 (time 1778614318.65 total 120.12 last 72.94)
 - open/close 8888 (time 1778614328.66 total 130.13 last 67.25)
 - open/close 9618 (time 1778614338.67 total 140.13 last 72.93)
total: 10000 open/close in 144.95 seconds: 68.99 ops/second
 - unlinked 0 (time 1778614348 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 768 (time 1778614447.68 total 10.01 last 76.71)
 - open/close 1555 (time 1778614457.69 total 20.02 last 78.64)
 - open/close 2387 (time 1778614467.69 total 30.02 last 83.17)
 - open/close 3260 (time 1778614477.70 total 40.03 last 87.22)
 - open/close 4048 (time 1778614487.71 total 50.04 last 78.73)
 - open/close 4941 (time 1778614497.71 total 60.04 last 89.28)
 - open/close 5808 (time 1778614507.72 total 70.05 last 86.64)
 - open/close 6544 (time 1778614517.73 total 80.06 last 73.52)
 - open/close 7306 (time 1778614527.73 total 90.07 last 76.17)
 - open/close 8051 (time 1778614537.74 total 100.07 last 74.44)
 - open/close 8750 (time 1778614547.75 total 110.08 last 69.82)
 - open/close 9459 (time 1778614557.76 total 120.09 last 70.84)
total: 10000 open/close in 126.53 seconds: 79.03 ops/second
 - unlinked 0 (time 1778614568 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 691 (time 1778614669.74 total 10.00 last 69.08)
 - open/close 1381 (time 1778614679.75 total 20.01 last 68.93)
 - open/close 2060 (time 1778614689.76 total 30.02 last 67.83)
 - open/close 2670 (time 1778614699.76 total 40.03 last 60.98)
 - open/close 3427 (time 1778614709.77 total 50.04 last 75.63)
 - open/close 4195 (time 1778614719.77 total 60.04 last 76.78)
 - open/close 4772 (time 1778614729.78 total 70.04 last 57.66)
 - open/close 5621 (time 1778614739.78 total 80.05 last 84.90)
 - open/close 6395 (time 1778614749.79 total 90.05 last 77.35)
 - open/close 7190 (time 1778614759.79 total 100.05 last 79.48)
 - open/close 7824 (time 1778614769.80 total 110.07 last 63.29)
 - open/close 8595 (time 1778614779.81 total 120.07 last 77.07)
 - open/close 9384 (time 1778614789.84 total 130.10 last 78.68)
total: 10000 open/close in 136.86 seconds: 73.07 ops/second
 - unlinked 0 (time 1778614801 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 813 (time 1778614906.98 total 10.01 last 81.18)
 - open/close 1578 (time 1778614916.98 total 20.02 last 76.47)
 - open/close 2365 (time 1778614926.99 total 30.03 last 78.64)
 - open/close 2992 (time 1778614936.99 total 40.03 last 62.70)
 - open/close 3586 (time 1778614946.99 total 50.03 last 59.38)
 - open/close 4253 (time 1778614957.00 total 60.04 last 66.63)
 - open/close 5013 (time 1778614967.01 total 70.05 last 75.93)
 - open/close 5694 (time 1778614977.02 total 80.06 last 68.05)
 - open/close 6344 (time 1778614987.03 total 90.07 last 64.93)
 - open/close 7010 (time 1778614997.03 total 100.07 last 66.60)
 - open/close 7688 (time 1778615007.03 total 110.07 last 67.79)
 - open/close 8436 (time 1778615017.07 total 120.11 last 74.53)
 - open/close 8979 (time 1778615027.07 total 130.11 last 54.28)
 - open/close 9631 (time 1778615037.08 total 140.12 last 65.14)
total: 10000 open/close in 146.68 seconds: 68.18 ops/second
 - unlinked 0 (time 1778615049 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 547 (time 1778615155.88 total 10.02 last 54.58)
 - open/close 1289 (time 1778615165.91 total 20.05 last 74.00)
 - open/close 1895 (time 1778615175.91 total 30.05 last 60.59)
 - open/close 2607 (time 1778615185.92 total 40.06 last 71.14)
 - open/close 3284 (time 1778615195.95 total 50.09 last 67.51)
 - open/close 3987 (time 1778615205.95 total 60.09 last 70.29)
 - open/close 4778 (time 1778615215.95 total 70.09 last 79.09)
 - open/close 5568 (time 1778615225.96 total 80.10 last 78.94)
 - open/close 6301 (time 1778615235.99 total 90.13 last 73.07)
 - open/close 6962 (time 1778615245.99 total 100.13 last 66.09)
 - open/close 7639 (time 1778615256.00 total 110.14 last 67.63)
 - open/close 8404 (time 1778615266.01 total 120.15 last 76.46)
 - open/close 9046 (time 1778615276.02 total 130.16 last 64.12)
 - open/close 9791 (time 1778615286.02 total 140.16 last 74.49)
total: 10000 open/close in 142.77 seconds: 70.04 ops/second
 - unlinked 0 (time 1778615293 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 619 (time 1778615407.65 total 10.02 last 61.77)
 - open/close 1253 (time 1778615417.67 total 20.04 last 63.29)
 - open/close 1832 (time 1778615427.69 total 30.06 last 57.78)
 - open/close 2296 (time 1778615437.69 total 40.07 last 46.36)
 - open/close 3085 (time 1778615447.70 total 50.07 last 78.84)
 - open/close 3803 (time 1778615457.71 total 60.08 last 71.76)
 - open/close 4431 (time 1778615467.72 total 70.09 last 62.75)
 - open/close 5168 (time 1778615477.73 total 80.10 last 73.63)
 - open/close 5875 (time 1778615487.73 total 90.10 last 70.69)
 - open/close 6546 (time 1778615497.74 total 100.12 last 67.00)
 - open/close 7236 (time 1778615507.75 total 110.12 last 68.99)
 - open/close 7972 (time 1778615517.76 total 120.13 last 73.49)
 - open/close 8724 (time 1778615527.77 total 130.14 last 75.10)
 - open/close 9433 (time 1778615537.77 total 140.15 last 70.88)
total: 10000 open/close in 147.43 seconds: 67.83 ops/second
 - unlinked 0 (time 1778615549 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 619 (time 1778615647.72 total 10.00 last 61.89)
 - open/close 1338 (time 1778615657.72 total 20.00 last 71.88)
 - open/close 1975 (time 1778615667.72 total 30.01 last 63.68)
 - open/close 2552 (time 1778615677.73 total 40.01 last 57.66)
 - open/close 3246 (time 1778615687.79 total 50.07 last 69.02)
 - open/close 3794 (time 1778615697.79 total 60.07 last 54.79)
 - open/close 4373 (time 1778615707.80 total 70.08 last 57.85)
 - open/close 5124 (time 1778615717.80 total 80.09 last 75.05)
 - open/close 6080 (time 1778615727.81 total 90.09 last 95.57)
 - open/close 6905 (time 1778615737.81 total 100.09 last 82.46)
 - open/close 7598 (time 1778615747.82 total 110.10 last 69.22)
 - open/close 8343 (time 1778615757.83 total 120.11 last 74.46)
 - open/close 8923 (time 1778615767.83 total 130.11 last 57.99)
 - open/close 9611 (time 1778615777.84 total 140.12 last 68.75)
total: 10000 open/close in 145.67 seconds: 68.65 ops/second
 - unlinked 0 (time 1778615787 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 597 (time 1778615892.96 total 10.01 last 59.66)
 - open/close 1372 (time 1778615902.96 total 20.01 last 77.45)
 - open/close 2087 (time 1778615912.97 total 30.02 last 71.47)
 - open/close 2898 (time 1778615922.97 total 40.02 last 81.10)
 - open/close 3710 (time 1778615932.97 total 50.02 last 81.15)
 - open/close 4472 (time 1778615942.99 total 60.04 last 76.08)
 - open/close 5220 (time 1778615953.01 total 70.06 last 74.66)
 - open/close 6001 (time 1778615963.01 total 80.07 last 78.04)
 - open/close 6707 (time 1778615973.03 total 90.08 last 70.51)
 - open/close 7392 (time 1778615983.03 total 100.08 last 68.50)
 - open/close 8122 (time 1778615993.03 total 110.08 last 72.97)
 - open/close 8954 (time 1778616003.04 total 120.09 last 83.12)
 - open/close 9753 (time 1778616013.07 total 130.12 last 79.69)
total: 10000 open/close in 133.75 seconds: 74.76 ops/second
 - unlinked 0 (time 1778616021 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 774 (time 1778616120.48 total 10.02 last 77.26)
 - open/close 1551 (time 1778616130.49 total 20.03 last 77.63)
 - open/close 2472 (time 1778616140.50 total 30.04 last 92.01)
 - open/close 3120 (time 1778616150.53 total 40.06 last 64.63)
 - open/close 3862 (time 1778616160.53 total 50.07 last 74.16)
 - open/close 4629 (time 1778616170.55 total 60.09 last 76.55)
 - open/close 5416 (time 1778616180.55 total 70.09 last 78.67)
 - open/close 6180 (time 1778616190.57 total 80.11 last 76.28)
 - open/close 6995 (time 1778616200.57 total 90.11 last 81.48)
 - open/close 7784 (time 1778616210.60 total 100.14 last 78.68)
 - open/close 8541 (time 1778616220.60 total 110.14 last 75.69)
 - open/close 9321 (time 1778616230.64 total 120.18 last 77.69)
total: 10000 open/close in 130.01 seconds: 76.92 ops/second
 - unlinked 0 (time 1778616244 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 748 (time 1778616344.44 total 10.00 last 74.80)
 - open/close 1678 (time 1778616354.44 total 20.00 last 92.96)
 - open/close 2433 (time 1778616364.45 total 30.01 last 75.43)
 - open/close 3185 (time 1778616374.46 total 40.02 last 75.13)
 - open/close 4048 (time 1778616384.47 total 50.03 last 86.22)
 - open/close 4726 (time 1778616394.49 total 60.06 last 67.62)
 - open/close 5408 (time 1778616404.54 total 70.10 last 67.90)
 - open/close 6080 (time 1778616414.55 total 80.11 last 67.15)
 - open/close 6733 (time 1778616424.55 total 90.12 last 65.28)
 - open/close 7613 (time 1778616434.56 total 100.12 last 87.96)
 - open/close 8682 (time 1778616444.56 total 110.12 last 106.87)
 - open/close 9492 (time 1778616454.57 total 120.13 last 80.94)
total: 10000 open/close in 129.37 seconds: 77.30 ops/second
 - unlinked 0 (time 1778616468 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
On OST0, 9837 used inodes rc=0
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.141@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server

unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: 
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2