== conf-sanity test 69: replace an OST with the same index ========================================================== 23:02:26 (1782529346)
start mds service on oleg244-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg244-server'
oleg244-server: oleg244-server.virtnet: executing load_modules_local
oleg244-server: Loading modules from /home/green/git/lustre-release/lustre
oleg244-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg244-server: detected 4 online CPUs by sysfs
oleg244-server: MODOPTS_LIBCFS=
oleg244-server: Force libcfs to create 2 CPU partitions
oleg244-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg244-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg244-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg244-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-OST0000
oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg244-client.virtnet:  -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 911 (time 1782529426.65 total 10.00 last 91.07)
 - open/close 1720 (time 1782529436.66 total 20.01 last 80.82)
 - open/close 2535 (time 1782529446.68 total 30.03 last 81.37)
 - open/close 3392 (time 1782529456.68 total 40.03 last 85.66)
 - open/close 4280 (time 1782529466.69 total 50.04 last 88.78)
 - open/close 5115 (time 1782529476.70 total 60.05 last 83.41)
 - open/close 5987 (time 1782529486.71 total 70.06 last 87.13)
 - open/close 6934 (time 1782529496.71 total 80.06 last 94.68)
 - open/close 7841 (time 1782529506.71 total 90.06 last 90.66)
 - open/close 8669 (time 1782529516.72 total 100.07 last 82.71)
 - open/close 9528 (time 1782529526.73 total 110.08 last 85.88)
total: 10000 open/close in 115.98 seconds: 86.22 ops/second
 - unlinked 0 (time 1782529537 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 775 (time 1782529624.98 total 10.01 last 77.46)
 - open/close 1526 (time 1782529634.98 total 20.01 last 75.10)
 - open/close 2450 (time 1782529644.99 total 30.02 last 92.30)
 - open/close 3268 (time 1782529655.00 total 40.03 last 81.70)
 - open/close 4138 (time 1782529665.01 total 50.04 last 86.93)
 - open/close 5182 (time 1782529675.02 total 60.05 last 104.30)
 - open/close 6199 (time 1782529685.03 total 70.05 last 101.64)
 - open/close 6993 (time 1782529695.03 total 80.06 last 79.35)
 - open/close 7845 (time 1782529705.05 total 90.08 last 85.05)
 - open/close 8760 (time 1782529715.05 total 100.08 last 91.47)
 - open/close 9601 (time 1782529725.06 total 110.08 last 84.05)
total: 10000 open/close in 114.27 seconds: 87.51 ops/second
 - unlinked 0 (time 1782529733 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 751 (time 1782529827.68 total 10.01 last 75.06)
 - open/close 1613 (time 1782529837.69 total 20.01 last 86.16)
 - open/close 2422 (time 1782529847.69 total 30.02 last 80.84)
 - open/close 3179 (time 1782529857.70 total 40.02 last 75.66)
 - open/close 3975 (time 1782529867.70 total 50.03 last 79.57)
 - open/close 4686 (time 1782529877.71 total 60.04 last 71.05)
 - open/close 5203 (time 1782529887.71 total 70.04 last 51.69)
 - open/close 5647 (time 1782529897.72 total 80.04 last 44.38)
 - open/close 6402 (time 1782529907.72 total 90.04 last 75.49)
 - open/close 7308 (time 1782529917.72 total 100.05 last 90.56)
 - open/close 8198 (time 1782529927.73 total 110.06 last 88.92)
 - open/close 8984 (time 1782529937.74 total 120.06 last 78.56)
 - open/close 9792 (time 1782529947.75 total 130.07 last 80.71)
total: 10000 open/close in 132.23 seconds: 75.63 ops/second
 - unlinked 0 (time 1782529954 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 943 (time 1782530045.14 total 10.01 last 94.23)
 - open/close 1662 (time 1782530055.14 total 20.01 last 71.88)
 - open/close 2429 (time 1782530065.17 total 30.03 last 76.53)
 - open/close 3258 (time 1782530075.19 total 40.05 last 82.72)
 - open/close 4032 (time 1782530085.19 total 50.06 last 77.37)
 - open/close 4861 (time 1782530095.20 total 60.07 last 82.84)
 - open/close 5613 (time 1782530105.21 total 70.07 last 75.14)
 - open/close 6193 (time 1782530115.22 total 80.09 last 57.92)
 - open/close 6906 (time 1782530125.23 total 90.10 last 71.20)
 - open/close 7552 (time 1782530135.24 total 100.11 last 64.53)
 - open/close 8289 (time 1782530145.25 total 110.12 last 73.65)
 - open/close 9087 (time 1782530155.26 total 120.13 last 79.72)
 - open/close 9975 (time 1782530165.26 total 130.13 last 88.79)
total: 10000 open/close in 130.40 seconds: 76.69 ops/second
 - unlinked 0 (time 1782530169 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 741 (time 1782530267.62 total 10.01 last 74.05)
 - open/close 1629 (time 1782530277.64 total 20.03 last 88.64)
 - open/close 2506 (time 1782530287.65 total 30.03 last 87.63)
 - open/close 3342 (time 1782530297.66 total 40.04 last 83.52)
 - open/close 4155 (time 1782530307.66 total 50.04 last 81.29)
 - open/close 5141 (time 1782530317.66 total 60.04 last 98.59)
 - open/close 6043 (time 1782530327.66 total 70.05 last 90.16)
 - open/close 6902 (time 1782530337.67 total 80.06 last 85.81)
 - open/close 7747 (time 1782530347.68 total 90.06 last 84.47)
 - open/close 8644 (time 1782530357.70 total 100.08 last 89.51)
 - open/close 9328 (time 1782530367.70 total 110.09 last 68.38)
total: 10000 open/close in 118.50 seconds: 84.39 ops/second
 - unlinked 0 (time 1782530380 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 790 (time 1782530478.62 total 10.01 last 78.96)
 - open/close 1623 (time 1782530488.63 total 20.01 last 83.23)
 - open/close 2375 (time 1782530498.63 total 30.02 last 75.18)
 - open/close 3183 (time 1782530508.63 total 40.02 last 80.79)
 - open/close 4016 (time 1782530518.63 total 50.02 last 83.29)
 - open/close 4946 (time 1782530528.64 total 60.03 last 92.92)
 - open/close 5781 (time 1782530538.66 total 70.04 last 83.38)
 - open/close 6623 (time 1782530548.68 total 80.07 last 84.02)
 - open/close 7496 (time 1782530558.68 total 90.07 last 87.24)
 - open/close 8214 (time 1782530568.69 total 100.08 last 71.75)
 - open/close 8754 (time 1782530578.70 total 110.09 last 53.95)
 - open/close 9459 (time 1782530588.71 total 120.10 last 70.42)
total: 10000 open/close in 126.97 seconds: 78.76 ops/second
 - unlinked 0 (time 1782530600 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 813 (time 1782530698.84 total 10.01 last 81.25)
 - open/close 1575 (time 1782530708.84 total 20.01 last 76.16)
 - open/close 2296 (time 1782530718.84 total 30.01 last 72.10)
 - open/close 2975 (time 1782530728.85 total 40.02 last 67.82)
 - open/close 3723 (time 1782530738.86 total 50.03 last 74.73)
 - open/close 4278 (time 1782530748.88 total 60.05 last 55.40)
 - open/close 5062 (time 1782530758.89 total 70.06 last 78.33)
 - open/close 5768 (time 1782530768.89 total 80.06 last 70.58)
 - open/close 6535 (time 1782530778.91 total 90.08 last 76.58)
 - open/close 7244 (time 1782530788.91 total 100.08 last 70.88)
 - open/close 8141 (time 1782530798.92 total 110.09 last 89.62)
 - open/close 8978 (time 1782530808.93 total 120.10 last 83.61)
 - open/close 9816 (time 1782530818.94 total 130.11 last 83.74)
total: 10000 open/close in 132.52 seconds: 75.46 ops/second
 - unlinked 0 (time 1782530826 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 599 (time 1782530926.35 total 10.01 last 59.86)
 - open/close 1476 (time 1782530936.36 total 20.02 last 87.60)
 - open/close 2395 (time 1782530946.37 total 30.03 last 91.78)
 - open/close 3192 (time 1782530956.38 total 40.04 last 79.60)
 - open/close 3959 (time 1782530966.41 total 50.07 last 76.51)
 - open/close 4696 (time 1782530976.42 total 60.08 last 73.63)
 - open/close 5624 (time 1782530986.42 total 70.08 last 92.76)
 - open/close 6492 (time 1782530996.44 total 80.10 last 86.65)
 - open/close 7407 (time 1782531006.44 total 90.10 last 91.48)
 - open/close 8251 (time 1782531016.45 total 100.10 last 84.38)
 - open/close 9066 (time 1782531026.45 total 110.11 last 81.44)
 - open/close 9880 (time 1782531036.45 total 120.11 last 81.40)
total: 10000 open/close in 121.38 seconds: 82.38 ops/second
 - unlinked 0 (time 1782531041 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 834 (time 1782531134.03 total 10.00 last 83.37)
 - open/close 1770 (time 1782531144.04 total 20.01 last 93.50)
 - open/close 2564 (time 1782531154.04 total 30.02 last 79.39)
 - open/close 3518 (time 1782531164.06 total 40.04 last 95.17)
 - open/close 4391 (time 1782531174.07 total 50.04 last 87.27)
 - open/close 5303 (time 1782531184.09 total 60.06 last 91.04)
 - open/close 6082 (time 1782531194.11 total 70.08 last 77.75)
 - open/close 6902 (time 1782531204.11 total 80.09 last 81.94)
 - open/close 7689 (time 1782531214.12 total 90.09 last 78.67)
 - open/close 8602 (time 1782531224.12 total 100.10 last 91.25)
 - open/close 9547 (time 1782531234.12 total 110.10 last 94.48)
total: 10000 open/close in 114.46 seconds: 87.37 ops/second
 - unlinked 0 (time 1782531242 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 817 (time 1782531334.58 total 10.01 last 81.64)
 - open/close 1641 (time 1782531344.59 total 20.01 last 82.36)
 - open/close 2610 (time 1782531354.59 total 30.01 last 96.88)
 - open/close 3471 (time 1782531364.59 total 40.01 last 86.09)
 - open/close 4328 (time 1782531374.60 total 50.02 last 85.65)
 - open/close 5181 (time 1782531384.61 total 60.04 last 85.18)
 - open/close 6052 (time 1782531394.62 total 70.04 last 87.05)
 - open/close 6919 (time 1782531404.63 total 80.06 last 86.58)
 - open/close 7766 (time 1782531414.64 total 90.06 last 84.62)
 - open/close 8634 (time 1782531424.66 total 100.08 last 86.67)
 - open/close 9511 (time 1782531434.66 total 110.09 last 87.64)
total: 10000 open/close in 115.34 seconds: 86.70 ops/second
 - unlinked 0 (time 1782531444 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg244-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg244-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.144@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.144@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg244-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-OST0000
oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg244-server: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg244-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg244-client.virtnet:  -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre
On OST0, 19918 used inodes rc=0
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg244-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg244-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.144@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.144@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.144@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.144@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.144@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg244-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-OST0000
oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg244-server: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg244-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg244-client.virtnet:  -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg244-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg244-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server
stop mds service on oleg244-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server

unloading modules via unload_modules_local on: 'oleg244-server'
oleg244-server: oleg244-server.virtnet: executing unload_modules_local
oleg244-server: 
modules unloaded.
pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2
pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2