== conf-sanity test 69: replace an OST with the same index ========================================================== 10:04:07 (1784124247)
start mds service on oleg328-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing load_modules_local
oleg328-server: Loading modules from /home/green/git/lustre-release/lustre
oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg328-server: detected 4 online CPUs by sysfs
oleg328-server: MODOPTS_LIBCFS=
oleg328-server: Force libcfs to create 2 CPU partitions
oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1679 (time 1784124295.99 total 10.00 last 167.84)
 - open/close 3207 (time 1784124306.00 total 20.01 last 152.72)
 - open/close 4943 (time 1784124316.00 total 30.01 last 173.58)
 - open/close 6509 (time 1784124326.01 total 40.01 last 156.54)
 - open/close 8199 (time 1784124336.01 total 50.02 last 168.91)
total: 10000 open/close in 59.68 seconds: 167.57 ops/second
 - unlinked 0 (time 1784124347 ; total 0 ; last 0)
total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second
 - open/close 1445 (time 1784124392.98 total 10.01 last 144.38)
 - open/close 2996 (time 1784124402.99 total 20.01 last 155.04)
 - open/close 4593 (time 1784124412.99 total 30.02 last 159.65)
 - open/close 5993 (time 1784124422.99 total 40.02 last 139.98)
 - open/close 7589 (time 1784124433.00 total 50.02 last 159.51)
 - open/close 8933 (time 1784124443.00 total 60.03 last 134.33)
total: 10000 open/close in 66.91 seconds: 149.46 ops/second
 - unlinked 0 (time 1784124451 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 670 (time 1784124544.32 total 10.00 last 66.97)
 - open/close 1465 (time 1784124554.32 total 20.01 last 79.49)
 - open/close 2266 (time 1784124564.33 total 30.01 last 80.06)
 - open/close 3134 (time 1784124574.33 total 40.01 last 86.77)
 - open/close 4040 (time 1784124584.34 total 50.02 last 90.56)
 - open/close 4950 (time 1784124594.34 total 60.02 last 90.95)
 - open/close 5816 (time 1784124604.35 total 70.03 last 86.55)
 - open/close 6752 (time 1784124614.35 total 80.03 last 93.57)
 - open/close 7691 (time 1784124624.35 total 90.04 last 93.85)
 - open/close 8590 (time 1784124634.36 total 100.04 last 89.85)
 - open/close 9450 (time 1784124644.36 total 110.05 last 85.98)
total: 10000 open/close in 116.49 seconds: 85.85 ops/second
 - unlinked 0 (time 1784124653 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 789 (time 1784124736.82 total 10.02 last 78.75)
 - open/close 1175 (time 1784124747.37 total 20.56 last 36.61)
 - open/close 1441 (time 1784124757.41 total 30.60 last 26.50)
 - open/close 1919 (time 1784124767.41 total 40.61 last 47.77)
 - open/close 2412 (time 1784124777.50 total 50.70 last 48.87)
 - open/close 2892 (time 1784124787.52 total 60.71 last 47.93)
 - open/close 3470 (time 1784124797.52 total 70.71 last 57.79)
 - open/close 4160 (time 1784124807.52 total 80.72 last 68.99)
 - open/close 4803 (time 1784124817.52 total 90.72 last 64.27)
 - open/close 5556 (time 1784124827.53 total 100.73 last 75.26)
 - open/close 6366 (time 1784124837.53 total 110.73 last 81.00)
 - open/close 7193 (time 1784124847.53 total 120.73 last 82.67)
 - open/close 8083 (time 1784124857.54 total 130.74 last 88.91)
 - open/close 8934 (time 1784124867.55 total 140.75 last 85.06)
 - open/close 9832 (time 1784124877.56 total 150.75 last 89.72)
total: 10000 open/close in 152.71 seconds: 65.49 ops/second
 - unlinked 0 (time 1784124882 ; total 0 ; last 0)
total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second
 - open/close 992 (time 1784124959.79 total 10.01 last 99.09)
 - open/close 1975 (time 1784124969.79 total 20.01 last 98.30)
 - open/close 2962 (time 1784124979.80 total 30.02 last 98.60)
 - open/close 4071 (time 1784124989.80 total 40.02 last 110.89)
 - open/close 5209 (time 1784124999.81 total 50.03 last 113.76)
 - open/close 6491 (time 1784125009.81 total 60.03 last 128.16)
 - open/close 7497 (time 1784125019.81 total 70.03 last 100.59)
 - open/close 8435 (time 1784125029.82 total 80.04 last 93.69)
 - open/close 9373 (time 1784125039.83 total 90.05 last 93.75)
total: 10000 open/close in 96.61 seconds: 103.51 ops/second
 - unlinked 0 (time 1784125049 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 907 (time 1784125123.67 total 10.01 last 90.65)
 - open/close 1917 (time 1784125133.68 total 20.01 last 100.94)
 - open/close 2896 (time 1784125143.68 total 30.02 last 97.86)
 - open/close 4066 (time 1784125153.69 total 40.02 last 116.93)
 - open/close 5165 (time 1784125163.69 total 50.02 last 109.88)
 - open/close 6206 (time 1784125173.69 total 60.03 last 104.04)
 - open/close 7266 (time 1784125183.70 total 70.03 last 105.99)
 - open/close 8270 (time 1784125193.70 total 80.04 last 100.33)
 - open/close 9261 (time 1784125203.71 total 90.04 last 99.06)
total: 10000 open/close in 98.03 seconds: 102.01 ops/second
 - unlinked 0 (time 1784125214 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 1241 (time 1784125285.20 total 10.01 last 124.03)
 - open/close 2382 (time 1784125295.20 total 20.01 last 114.09)
 - open/close 3503 (time 1784125305.21 total 30.01 last 112.00)
 - open/close 4707 (time 1784125315.21 total 40.02 last 120.37)
 - open/close 5828 (time 1784125325.22 total 50.02 last 112.04)
 - open/close 6929 (time 1784125335.22 total 60.03 last 110.05)
 - open/close 8224 (time 1784125345.23 total 70.03 last 129.40)
 - open/close 9383 (time 1784125355.23 total 80.04 last 115.89)
total: 10000 open/close in 85.85 seconds: 116.48 ops/second
 - unlinked 0 (time 1784125363 ; total 0 ; last 0)
total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second
 - open/close 1013 (time 1784125427.48 total 10.00 last 101.29)
 - open/close 2129 (time 1784125437.48 total 20.00 last 111.59)
 - open/close 3230 (time 1784125447.48 total 30.01 last 110.07)
 - open/close 4462 (time 1784125457.48 total 40.01 last 123.19)
 - open/close 5666 (time 1784125467.49 total 50.01 last 120.34)
 - open/close 6811 (time 1784125477.49 total 60.02 last 114.45)
 - open/close 8032 (time 1784125487.50 total 70.02 last 122.04)
 - open/close 9135 (time 1784125497.50 total 80.02 last 110.28)
total: 10000 open/close in 88.62 seconds: 112.85 ops/second
 - unlinked 0 (time 1784125508 ; total 0 ; last 0)
total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second
 - open/close 1200 (time 1784125577.32 total 10.00 last 119.98)
 - open/close 2412 (time 1784125587.34 total 20.02 last 120.93)
 - open/close 3616 (time 1784125597.35 total 30.03 last 120.36)
 - open/close 4763 (time 1784125607.35 total 40.03 last 114.62)
 - open/close 6088 (time 1784125617.36 total 50.04 last 132.45)
 - open/close 7333 (time 1784125627.36 total 60.04 last 124.43)
 - open/close 8601 (time 1784125637.37 total 70.05 last 126.78)
total: 10000 open/close in 79.01 seconds: 126.57 ops/second
 - unlinked 0 (time 1784125648 ; total 0 ; last 0)
total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second
 - open/close 1253 (time 1784125701.51 total 10.00 last 125.30)
 - open/close 2526 (time 1784125711.51 total 20.00 last 127.26)
 - open/close 3720 (time 1784125721.51 total 30.00 last 119.40)
 - open/close 5058 (time 1784125731.52 total 40.01 last 133.77)
 - open/close 6307 (time 1784125741.52 total 50.01 last 124.82)
 - open/close 7741 (time 1784125751.52 total 60.01 last 143.38)
 - open/close 8977 (time 1784125761.53 total 70.02 last 123.58)
total: 10000 open/close in 77.44 seconds: 129.14 ops/second
 - unlinked 0 (time 1784125770 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
On OST0, 19430 used inodes rc=0
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.128@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.128@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.128@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server
stop mds service on oleg328-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server

unloading modules via unload_modules_local on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing unload_modules_local
oleg328-server: 
modules unloaded.
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2