== conf-sanity test 69: replace an OST with the same index ========================================================== 10:56:27 (1787842587)
start mds service on oleg231-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg231-server'
oleg231-server: oleg231-server.virtnet: executing load_modules_local
oleg231-server: Loading modules from /home/green/git/lustre-release/lustre
oleg231-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg231-server: detected 4 online CPUs by sysfs
oleg231-server: MODOPTS_LIBCFS=
oleg231-server: Force libcfs to create 2 CPU partitions
oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg231-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg231-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-OST0000
oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg231-client.virtnet:  -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 907 (time 1787842674.42 total 10.01 last 90.63)
 - open/close 1694 (time 1787842684.43 total 20.01 last 78.67)
 - open/close 2523 (time 1787842694.43 total 30.02 last 82.86)
 - open/close 3403 (time 1787842704.44 total 40.02 last 87.94)
 - open/close 4274 (time 1787842714.45 total 50.03 last 87.01)
 - open/close 5144 (time 1787842724.45 total 60.04 last 86.94)
 - open/close 5931 (time 1787842734.46 total 70.04 last 78.69)
 - open/close 6632 (time 1787842744.47 total 80.05 last 70.02)
 - open/close 7421 (time 1787842754.48 total 90.06 last 78.82)
 - open/close 8136 (time 1787842764.48 total 100.07 last 71.47)
 - open/close 8930 (time 1787842774.49 total 110.08 last 79.33)
 - open/close 9776 (time 1787842784.51 total 120.09 last 84.47)
total: 10000 open/close in 122.68 seconds: 81.51 ops/second
 - unlinked 0 (time 1787842791 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 747 (time 1787842889.22 total 10.01 last 74.65)
 - open/close 1460 (time 1787842899.22 total 20.01 last 71.30)
 - open/close 2260 (time 1787842909.23 total 30.02 last 79.89)
 - open/close 3120 (time 1787842919.24 total 40.02 last 85.99)
 - open/close 3883 (time 1787842929.24 total 50.03 last 76.26)
 - open/close 4660 (time 1787842939.25 total 60.04 last 77.63)
 - open/close 5468 (time 1787842949.25 total 70.04 last 80.79)
 - open/close 6220 (time 1787842959.28 total 80.07 last 74.95)
 - open/close 6908 (time 1787842969.29 total 90.08 last 68.76)
 - open/close 7777 (time 1787842979.30 total 100.09 last 86.78)
 - open/close 8573 (time 1787842989.31 total 110.10 last 79.54)
 - open/close 9359 (time 1787842999.32 total 120.11 last 78.54)
total: 10000 open/close in 127.40 seconds: 78.49 ops/second
 - unlinked 0 (time 1787843011 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 770 (time 1787843108.40 total 10.01 last 76.94)
 - open/close 1566 (time 1787843118.41 total 20.02 last 79.53)
 - open/close 2352 (time 1787843128.42 total 30.02 last 78.54)
 - open/close 3215 (time 1787843138.43 total 40.03 last 86.23)
 - open/close 4067 (time 1787843148.43 total 50.04 last 85.16)
 - open/close 4954 (time 1787843158.44 total 60.04 last 88.63)
 - open/close 5700 (time 1787843168.45 total 70.05 last 74.54)
 - open/close 6496 (time 1787843178.48 total 80.09 last 79.32)
 - open/close 7361 (time 1787843188.50 total 90.10 last 86.39)
 - open/close 8127 (time 1787843198.50 total 100.10 last 76.56)
 - open/close 8864 (time 1787843208.51 total 110.11 last 73.64)
 - open/close 9644 (time 1787843218.52 total 120.12 last 77.92)
total: 10000 open/close in 124.34 seconds: 80.42 ops/second
 - unlinked 0 (time 1787843227 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 860 (time 1787843318.97 total 10.00 last 86.00)
 - open/close 1594 (time 1787843328.98 total 20.01 last 73.33)
 - open/close 2430 (time 1787843338.99 total 30.02 last 83.52)
 - open/close 3147 (time 1787843348.99 total 40.02 last 71.67)
 - open/close 4080 (time 1787843359.00 total 50.03 last 93.22)
 - open/close 4859 (time 1787843369.00 total 60.04 last 77.86)
 - open/close 5595 (time 1787843379.01 total 70.04 last 73.58)
 - open/close 6294 (time 1787843389.01 total 80.04 last 69.86)
 - open/close 7160 (time 1787843399.01 total 90.05 last 86.58)
 - open/close 7985 (time 1787843409.03 total 100.06 last 82.39)
 - open/close 8808 (time 1787843419.07 total 110.11 last 81.92)
 - open/close 9591 (time 1787843429.08 total 120.11 last 78.27)
total: 10000 open/close in 125.87 seconds: 79.44 ops/second
 - unlinked 0 (time 1787843439 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 892 (time 1787843531.93 total 10.00 last 89.17)
 - open/close 1700 (time 1787843541.93 total 20.00 last 80.79)
 - open/close 2548 (time 1787843551.94 total 30.01 last 84.73)
 - open/close 3439 (time 1787843561.94 total 40.01 last 89.09)
 - open/close 4369 (time 1787843571.96 total 50.03 last 92.81)
 - open/close 5200 (time 1787843581.96 total 60.04 last 83.08)
 - open/close 6064 (time 1787843591.97 total 70.04 last 86.39)
 - open/close 6918 (time 1787843601.98 total 80.05 last 85.28)
 - open/close 7738 (time 1787843611.98 total 90.06 last 81.98)
 - open/close 8582 (time 1787843621.99 total 100.06 last 84.36)
 - open/close 9219 (time 1787843631.99 total 110.07 last 63.66)
 - open/close 9918 (time 1787843642.00 total 120.08 last 69.83)
total: 10000 open/close in 121.03 seconds: 82.62 ops/second
 - unlinked 0 (time 1787843646 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 747 (time 1787843741.34 total 10.00 last 74.68)
 - open/close 1580 (time 1787843751.34 total 20.00 last 83.30)
 - open/close 2356 (time 1787843761.34 total 30.01 last 77.59)
 - open/close 3137 (time 1787843771.37 total 40.03 last 77.89)
 - open/close 3894 (time 1787843781.37 total 50.04 last 75.67)
 - open/close 4693 (time 1787843791.38 total 60.04 last 79.85)
 - open/close 5528 (time 1787843801.38 total 70.05 last 83.47)
 - open/close 6282 (time 1787843811.38 total 80.05 last 75.40)
 - open/close 7044 (time 1787843821.40 total 90.06 last 76.07)
 - open/close 7786 (time 1787843831.42 total 100.08 last 74.09)
 - open/close 8566 (time 1787843841.42 total 110.08 last 77.98)
 - open/close 9416 (time 1787843851.42 total 120.08 last 84.99)
total: 10000 open/close in 127.64 seconds: 78.34 ops/second
 - unlinked 0 (time 1787843863 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 808 (time 1787843963.85 total 10.01 last 80.73)
 - open/close 1613 (time 1787843973.86 total 20.02 last 80.44)
 - open/close 2399 (time 1787843983.86 total 30.02 last 78.58)
 - open/close 3117 (time 1787843993.87 total 40.02 last 71.76)
 - open/close 3886 (time 1787844003.88 total 50.04 last 76.78)
 - open/close 4660 (time 1787844013.89 total 60.05 last 77.32)
 - open/close 5411 (time 1787844023.90 total 70.06 last 75.05)
 - open/close 6166 (time 1787844033.91 total 80.07 last 75.38)
 - open/close 7035 (time 1787844043.92 total 90.08 last 86.83)
 - open/close 7806 (time 1787844053.97 total 100.13 last 76.70)
 - open/close 8566 (time 1787844063.97 total 110.13 last 76.00)
 - open/close 9517 (time 1787844073.98 total 120.14 last 95.03)
total: 10000 open/close in 125.59 seconds: 79.62 ops/second
 - unlinked 0 (time 1787844083 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 813 (time 1787844168.88 total 10.00 last 81.28)
 - open/close 1697 (time 1787844178.89 total 20.01 last 88.34)
 - open/close 2471 (time 1787844188.90 total 30.02 last 77.31)
 - open/close 3313 (time 1787844198.91 total 40.03 last 84.12)
 - open/close 4174 (time 1787844208.91 total 50.03 last 86.09)
 - open/close 5022 (time 1787844218.92 total 60.04 last 84.70)
 - open/close 5970 (time 1787844228.93 total 70.05 last 94.76)
 - open/close 6857 (time 1787844238.93 total 80.05 last 88.68)
 - open/close 7560 (time 1787844248.94 total 90.06 last 70.21)
 - open/close 8320 (time 1787844258.95 total 100.07 last 75.94)
 - open/close 9151 (time 1787844268.98 total 110.10 last 82.86)
total: 10000 open/close in 119.68 seconds: 83.56 ops/second
 - unlinked 0 (time 1787844282 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 652 (time 1787844396.03 total 10.03 last 65.00)
 - open/close 1350 (time 1787844406.04 total 20.04 last 69.76)
 - open/close 2029 (time 1787844416.04 total 30.04 last 67.86)
 - open/close 2654 (time 1787844426.05 total 40.04 last 62.49)
 - open/close 3274 (time 1787844436.06 total 50.05 last 61.94)
 - open/close 3835 (time 1787844446.07 total 60.07 last 56.03)
 - open/close 4511 (time 1787844456.08 total 70.08 last 67.54)
 - open/close 5351 (time 1787844466.08 total 80.08 last 83.95)
 - open/close 6196 (time 1787844476.09 total 90.08 last 84.49)
 - open/close 7125 (time 1787844486.09 total 100.09 last 92.84)
 - open/close 7983 (time 1787844496.11 total 110.10 last 85.68)
 - open/close 8564 (time 1787844506.11 total 120.11 last 58.07)
 - open/close 9408 (time 1787844516.11 total 130.11 last 84.40)
total: 10000 open/close in 138.32 seconds: 72.29 ops/second
 - unlinked 0 (time 1787844528 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 731 (time 1787844618.97 total 10.00 last 73.10)
 - open/close 1536 (time 1787844628.97 total 20.00 last 80.49)
 - open/close 2316 (time 1787844638.98 total 30.02 last 77.87)
 - open/close 3155 (time 1787844648.99 total 40.03 last 83.83)
 - open/close 4006 (time 1787844659.02 total 50.05 last 84.90)
 - open/close 4846 (time 1787844669.02 total 60.06 last 83.95)
 - open/close 5655 (time 1787844679.03 total 70.06 last 80.85)
 - open/close 6409 (time 1787844689.04 total 80.07 last 75.32)
 - open/close 7249 (time 1787844699.07 total 90.11 last 83.73)
 - open/close 7955 (time 1787844709.08 total 100.11 last 70.58)
 - open/close 8705 (time 1787844719.08 total 110.11 last 74.96)
 - open/close 9490 (time 1787844729.09 total 120.12 last 78.44)
total: 10000 open/close in 126.52 seconds: 79.04 ops/second
 - unlinked 0 (time 1787844740 ; total 1 ; last 1)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg231-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg231-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg231-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-OST0000
oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg231-client.virtnet:  -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg231-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg231-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.131@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg231-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-OST0000
oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg231-client.virtnet:  -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg231-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg231-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server
stop mds service on oleg231-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server

unloading modules via unload_modules_local on: 'oleg231-server'
oleg231-server: oleg231-server.virtnet: executing unload_modules_local
oleg231-server: 
modules unloaded.
pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2
pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2