== conf-sanity test 69: replace an OST with the same index ========================================================== 16:17:28 (1787602648)
start mds service on oleg642-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg642-server'
oleg642-server: oleg642-server.virtnet: executing load_modules_local
oleg642-server: Loading modules from /home/green/git/lustre-release/lustre
oleg642-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg642-server: detected 4 online CPUs by sysfs
oleg642-server: MODOPTS_LIBCFS=
oleg642-server: Force libcfs to create 2 CPU partitions
oleg642-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg642-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg642-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg642-server: oleg642-server.virtnet: executing set_default_debug -1 all
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg642-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg642-server: oleg642-server.virtnet: executing set_default_debug -1 all
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg642-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg642-server: oleg642-server.virtnet: executing set_default_debug -1 all
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1
Started lustre-OST0000
oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg642-client.virtnet:  -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 793 (time 1787602746.65 total 10.02 last 79.13)
 - open/close 1537 (time 1787602756.65 total 20.03 last 74.34)
 - open/close 2284 (time 1787602766.69 total 30.06 last 74.46)
 - open/close 3004 (time 1787602776.70 total 40.07 last 71.92)
 - open/close 3737 (time 1787602786.71 total 50.08 last 73.21)
 - open/close 4507 (time 1787602796.72 total 60.10 last 76.90)
 - open/close 5304 (time 1787602806.73 total 70.11 last 79.62)
 - open/close 6092 (time 1787602816.74 total 80.12 last 78.71)
 - open/close 6880 (time 1787602826.75 total 90.13 last 78.72)
 - open/close 7670 (time 1787602836.76 total 100.13 last 78.96)
 - open/close 8446 (time 1787602846.76 total 110.14 last 77.55)
 - open/close 9207 (time 1787602856.77 total 120.15 last 76.04)
 - open/close 9890 (time 1787602866.81 total 130.19 last 68.02)
total: 10000 open/close in 131.61 seconds: 75.98 ops/second
 - unlinked 0 (time 1787602872 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 634 (time 1787602966.22 total 10.00 last 63.38)
 - open/close 1254 (time 1787602976.23 total 20.01 last 61.97)
 - open/close 1975 (time 1787602986.23 total 30.01 last 72.08)
 - open/close 2732 (time 1787602996.24 total 40.02 last 75.65)
 - open/close 3481 (time 1787603006.27 total 50.05 last 74.65)
 - open/close 4209 (time 1787603016.28 total 60.06 last 72.76)
 - open/close 4969 (time 1787603026.29 total 70.08 last 75.86)
 - open/close 5728 (time 1787603036.30 total 80.08 last 75.90)
 - open/close 6430 (time 1787603046.31 total 90.09 last 70.10)
 - open/close 7146 (time 1787603056.31 total 100.10 last 71.56)
 - open/close 7906 (time 1787603066.33 total 110.11 last 75.91)
 - open/close 8585 (time 1787603076.33 total 120.11 last 67.87)
 - open/close 9363 (time 1787603086.34 total 130.12 last 77.76)
total: 10000 open/close in 138.85 seconds: 72.02 ops/second
 - unlinked 0 (time 1787603099 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 670 (time 1787603197.28 total 10.01 last 66.92)
 - open/close 1286 (time 1787603207.28 total 20.01 last 61.59)
 - open/close 1956 (time 1787603217.28 total 30.02 last 66.99)
 - open/close 2661 (time 1787603227.29 total 40.02 last 70.45)
 - open/close 3441 (time 1787603237.29 total 50.03 last 77.96)
 - open/close 4187 (time 1787603247.30 total 60.03 last 74.56)
 - open/close 4899 (time 1787603257.30 total 70.03 last 71.18)
 - open/close 5642 (time 1787603267.30 total 80.04 last 74.27)
 - open/close 6370 (time 1787603277.31 total 90.05 last 72.74)
 - open/close 7047 (time 1787603287.34 total 100.07 last 67.53)
 - open/close 7654 (time 1787603297.34 total 110.08 last 60.69)
 - open/close 8308 (time 1787603307.35 total 120.08 last 65.35)
 - open/close 9007 (time 1787603317.36 total 130.09 last 69.85)
 - open/close 9713 (time 1787603327.36 total 140.10 last 70.56)
total: 10000 open/close in 144.09 seconds: 69.40 ops/second
 - unlinked 0 (time 1787603335 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 672 (time 1787603433.87 total 10.01 last 67.14)
 - open/close 1412 (time 1787603443.88 total 20.02 last 73.92)
 - open/close 2148 (time 1787603453.89 total 30.03 last 73.52)
 - open/close 2766 (time 1787603463.90 total 40.04 last 61.74)
 - open/close 3425 (time 1787603473.91 total 50.05 last 65.85)
 - open/close 4092 (time 1787603483.91 total 60.05 last 66.65)
 - open/close 4726 (time 1787603493.92 total 70.06 last 63.38)
 - open/close 5403 (time 1787603503.92 total 80.06 last 67.69)
 - open/close 6151 (time 1787603513.92 total 90.06 last 74.76)
 - open/close 6879 (time 1787603523.93 total 100.07 last 72.74)
 - open/close 7614 (time 1787603533.94 total 110.08 last 73.46)
 - open/close 8281 (time 1787603543.99 total 120.12 last 66.38)
 - open/close 8970 (time 1787603553.99 total 130.13 last 68.86)
 - open/close 9616 (time 1787603563.99 total 140.13 last 64.59)
total: 10000 open/close in 145.91 seconds: 68.53 ops/second
 - unlinked 0 (time 1787603574 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 759 (time 1787603671.43 total 10.01 last 75.81)
 - open/close 1498 (time 1787603681.45 total 20.02 last 73.80)
 - open/close 2178 (time 1787603691.45 total 30.03 last 67.98)
 - open/close 2864 (time 1787603701.48 total 40.06 last 68.36)
 - open/close 3582 (time 1787603711.50 total 50.08 last 71.67)
 - open/close 4144 (time 1787603721.52 total 60.10 last 56.07)
 - open/close 4798 (time 1787603731.54 total 70.12 last 65.28)
 - open/close 5408 (time 1787603741.56 total 80.14 last 60.88)
 - open/close 6119 (time 1787603751.56 total 90.14 last 71.10)
 - open/close 6808 (time 1787603761.57 total 100.15 last 68.86)
 - open/close 7510 (time 1787603771.57 total 110.15 last 70.19)
 - open/close 8226 (time 1787603781.57 total 120.15 last 71.57)
 - open/close 8981 (time 1787603791.58 total 130.15 last 75.48)
 - open/close 9767 (time 1787603801.58 total 140.16 last 78.57)
total: 10000 open/close in 142.74 seconds: 70.06 ops/second
 - unlinked 0 (time 1787603807 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 777 (time 1787603905.67 total 10.02 last 77.57)
 - open/close 1493 (time 1787603915.68 total 20.02 last 71.55)
 - open/close 2276 (time 1787603925.69 total 30.03 last 78.25)
 - open/close 3061 (time 1787603935.69 total 40.03 last 78.49)
 - open/close 3733 (time 1787603945.69 total 50.04 last 67.17)
 - open/close 4256 (time 1787603955.69 total 60.04 last 52.29)
 - open/close 4857 (time 1787603965.70 total 70.05 last 60.04)
 - open/close 5461 (time 1787603975.71 total 80.06 last 60.36)
 - open/close 6017 (time 1787603985.73 total 90.08 last 55.47)
 - open/close 6689 (time 1787603995.74 total 100.08 last 67.18)
 - open/close 7441 (time 1787604005.75 total 110.09 last 75.11)
 - open/close 8214 (time 1787604015.76 total 120.10 last 77.23)
 - open/close 8977 (time 1787604025.77 total 130.11 last 76.22)
 - open/close 9689 (time 1787604035.77 total 140.12 last 71.18)
total: 10000 open/close in 144.92 seconds: 69.00 ops/second
 - unlinked 0 (time 1787604044 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 725 (time 1787604140.38 total 10.01 last 72.46)
 - open/close 1423 (time 1787604150.41 total 20.04 last 69.54)
 - open/close 2132 (time 1787604160.42 total 30.05 last 70.86)
 - open/close 2832 (time 1787604170.42 total 40.05 last 69.99)
 - open/close 3630 (time 1787604180.43 total 50.05 last 79.76)
 - open/close 4439 (time 1787604190.43 total 60.06 last 80.89)
 - open/close 5098 (time 1787604200.43 total 70.06 last 65.87)
 - open/close 5914 (time 1787604210.44 total 80.07 last 81.55)
 - open/close 6620 (time 1787604220.46 total 90.09 last 70.47)
 - open/close 7306 (time 1787604230.46 total 100.09 last 68.60)
 - open/close 7967 (time 1787604240.47 total 110.10 last 65.98)
 - open/close 8660 (time 1787604250.48 total 120.11 last 69.27)
 - open/close 9299 (time 1787604260.49 total 130.12 last 63.85)
total: 10000 open/close in 139.04 seconds: 71.92 ops/second
 - unlinked 0 (time 1787604273 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 786 (time 1787604372.68 total 10.02 last 78.43)
 - open/close 1583 (time 1787604382.69 total 20.03 last 79.61)
 - open/close 2322 (time 1787604392.70 total 30.04 last 73.82)
 - open/close 3089 (time 1787604402.71 total 40.05 last 76.66)
 - open/close 3774 (time 1787604412.71 total 50.05 last 68.48)
 - open/close 4514 (time 1787604422.72 total 60.06 last 73.96)
 - open/close 5234 (time 1787604432.73 total 70.07 last 71.89)
 - open/close 6009 (time 1787604442.74 total 80.08 last 77.40)
 - open/close 6717 (time 1787604452.75 total 90.09 last 70.77)
 - open/close 7383 (time 1787604462.77 total 100.11 last 66.50)
 - open/close 8005 (time 1787604472.78 total 110.12 last 62.12)
 - open/close 8708 (time 1787604482.78 total 120.12 last 70.28)
 - open/close 9459 (time 1787604492.79 total 130.13 last 75.03)
total: 10000 open/close in 136.93 seconds: 73.03 ops/second
 - unlinked 0 (time 1787604505 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 707 (time 1787604605.02 total 10.01 last 70.62)
 - open/close 1449 (time 1787604615.04 total 20.03 last 74.05)
 - open/close 2231 (time 1787604625.05 total 30.04 last 78.14)
 - open/close 2942 (time 1787604635.06 total 40.04 last 71.07)
 - open/close 3545 (time 1787604645.07 total 50.06 last 60.22)
 - open/close 4174 (time 1787604655.10 total 60.09 last 62.69)
 - open/close 4766 (time 1787604665.14 total 70.13 last 58.98)
 - open/close 5571 (time 1787604675.15 total 80.14 last 80.43)
 - open/close 6292 (time 1787604685.15 total 90.14 last 72.10)
 - open/close 6915 (time 1787604695.15 total 100.14 last 62.26)
 - open/close 7447 (time 1787604705.16 total 110.15 last 53.17)
 - open/close 8159 (time 1787604715.16 total 120.15 last 71.19)
 - open/close 8858 (time 1787604725.16 total 130.15 last 69.90)
 - open/close 9491 (time 1787604735.16 total 140.15 last 63.29)
total: 10000 open/close in 148.29 seconds: 67.44 ops/second
 - unlinked 0 (time 1787604748 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 713 (time 1787604848.44 total 10.01 last 71.24)
 - open/close 1420 (time 1787604858.45 total 20.02 last 70.63)
 - open/close 2058 (time 1787604868.46 total 30.03 last 63.72)
 - open/close 2796 (time 1787604878.46 total 40.04 last 73.74)
 - open/close 3552 (time 1787604888.47 total 50.04 last 75.58)
 - open/close 4235 (time 1787604898.48 total 60.05 last 68.23)
 - open/close 4999 (time 1787604908.48 total 70.05 last 76.37)
 - open/close 5588 (time 1787604918.49 total 80.06 last 58.85)
 - open/close 6329 (time 1787604928.49 total 90.07 last 74.07)
 - open/close 7088 (time 1787604938.50 total 100.07 last 75.87)
 - open/close 7813 (time 1787604948.50 total 110.08 last 72.48)
 - open/close 8536 (time 1787604958.51 total 120.08 last 72.23)
 - open/close 9298 (time 1787604968.53 total 130.10 last 76.08)
total: 10000 open/close in 139.68 seconds: 71.59 ops/second
 - unlinked 0 (time 1787604983 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg642-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg642-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg642-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.142@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.142@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg642-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg642-server: oleg642-server.virtnet: executing set_default_debug -1 all
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1
Started lustre-OST0000
oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg642-server: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg642-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg642-server: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg642-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg642-client.virtnet:  -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre
On OST0, 20520 used inodes rc=0
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg642-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg642-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg642-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.142@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.142@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.142@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.142@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.206.142@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg642-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg642-server: oleg642-server.virtnet: executing set_default_debug -1 all
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1
Started lustre-OST0000
oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg642-server: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg642-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec
oleg642-server: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg642-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg642-client.virtnet:  -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg642-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg642-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg642-server
stop mds service on oleg642-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg642-server
stop mds service on oleg642-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg642-server

unloading modules via unload_modules_local on: 'oleg642-server'
oleg642-server: oleg642-server.virtnet: executing unload_modules_local
oleg642-server: 
modules unloaded.
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 2
pdsh@oleg642-client: oleg642-client: ssh exited with exit code 2
pdsh@oleg642-client: oleg642-server: ssh exited with exit code 2
pdsh@oleg642-client: oleg642-client: ssh exited with exit code 2