== conf-sanity test 69: replace an OST with the same index ========================================================== 15:25:00 (1787599500)
start mds service on oleg625-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg625-server'
oleg625-server: oleg625-server.virtnet: executing load_modules_local
oleg625-server: Loading modules from /home/green/git/lustre-release/lustre
oleg625-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg625-server: detected 4 online CPUs by sysfs
oleg625-server: MODOPTS_LIBCFS=
oleg625-server: Force libcfs to create 2 CPU partitions
oleg625-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg625-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg625-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg625-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1
Started lustre-OST0000
oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg625-client.virtnet:  -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 917 (time 1787599582.48 total 10.01 last 91.65)
 - open/close 1744 (time 1787599592.49 total 20.02 last 82.62)
 - open/close 2552 (time 1787599602.50 total 30.03 last 80.72)
 - open/close 3256 (time 1787599612.50 total 40.03 last 70.37)
 - open/close 4149 (time 1787599622.51 total 50.03 last 89.26)
 - open/close 4978 (time 1787599632.51 total 60.04 last 82.86)
 - open/close 5803 (time 1787599642.51 total 70.04 last 82.50)
 - open/close 6699 (time 1787599652.51 total 80.04 last 89.58)
 - open/close 7546 (time 1787599662.52 total 90.04 last 84.68)
 - open/close 8428 (time 1787599672.52 total 100.05 last 88.18)
 - open/close 9288 (time 1787599682.53 total 110.05 last 85.94)
total: 10000 open/close in 118.83 seconds: 84.15 ops/second
 - unlinked 0 (time 1787599695 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 669 (time 1787599780.77 total 10.01 last 66.85)
 - open/close 1372 (time 1787599790.77 total 20.01 last 70.28)
 - open/close 1981 (time 1787599800.77 total 30.01 last 60.89)
 - open/close 2822 (time 1787599810.77 total 40.01 last 84.09)
 - open/close 3680 (time 1787599820.78 total 50.02 last 85.70)
 - open/close 4559 (time 1787599830.79 total 60.03 last 87.82)
 - open/close 5442 (time 1787599840.80 total 70.04 last 88.27)
 - open/close 6286 (time 1787599850.80 total 80.04 last 84.37)
 - open/close 7239 (time 1787599860.83 total 90.07 last 95.00)
 - open/close 7985 (time 1787599870.84 total 100.08 last 74.53)
 - open/close 8844 (time 1787599880.85 total 110.09 last 85.83)
 - open/close 9719 (time 1787599890.85 total 120.09 last 87.45)
total: 10000 open/close in 123.58 seconds: 80.92 ops/second
 - unlinked 0 (time 1787599898 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 803 (time 1787599984.54 total 10.01 last 80.22)
 - open/close 1456 (time 1787599994.54 total 20.02 last 65.27)
 - open/close 2355 (time 1787600004.54 total 30.02 last 89.89)
 - open/close 3293 (time 1787600014.55 total 40.02 last 93.75)
 - open/close 4265 (time 1787600024.55 total 50.03 last 97.14)
 - open/close 5163 (time 1787600034.56 total 60.03 last 89.76)
 - open/close 5999 (time 1787600044.57 total 70.04 last 83.51)
 - open/close 6862 (time 1787600054.58 total 80.06 last 86.15)
 - open/close 7772 (time 1787600064.58 total 90.06 last 91.00)
 - open/close 8609 (time 1787600074.59 total 100.06 last 83.68)
 - open/close 9436 (time 1787600084.59 total 110.06 last 82.69)
total: 10000 open/close in 116.80 seconds: 85.61 ops/second
 - unlinked 0 (time 1787600094 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 909 (time 1787600187.20 total 10.01 last 90.79)
 - open/close 1797 (time 1787600197.20 total 20.02 last 88.74)
 - open/close 2624 (time 1787600207.21 total 30.02 last 82.67)
 - open/close 3534 (time 1787600217.22 total 40.03 last 90.89)
 - open/close 4388 (time 1787600227.22 total 50.04 last 85.38)
 - open/close 5293 (time 1787600237.23 total 60.04 last 90.47)
 - open/close 6231 (time 1787600247.23 total 70.04 last 93.79)
 - open/close 7053 (time 1787600257.23 total 80.05 last 82.16)
 - open/close 7927 (time 1787600267.24 total 90.05 last 87.33)
 - open/close 8798 (time 1787600277.24 total 100.06 last 87.08)
 - open/close 9644 (time 1787600287.24 total 110.06 last 84.60)
total: 10000 open/close in 114.09 seconds: 87.65 ops/second
 - unlinked 0 (time 1787600294 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 747 (time 1787600381.87 total 10.00 last 74.69)
 - open/close 1619 (time 1787600391.88 total 20.01 last 87.13)
 - open/close 2542 (time 1787600401.89 total 30.02 last 92.19)
 - open/close 3394 (time 1787600411.91 total 40.04 last 85.01)
 - open/close 4338 (time 1787600421.92 total 50.05 last 94.30)
 - open/close 5227 (time 1787600431.93 total 60.06 last 88.88)
 - open/close 6098 (time 1787600441.93 total 70.06 last 87.05)
 - open/close 7027 (time 1787600451.93 total 80.06 last 92.87)
 - open/close 7908 (time 1787600461.95 total 90.08 last 87.95)
 - open/close 8789 (time 1787600471.97 total 100.10 last 87.95)
 - open/close 9673 (time 1787600481.98 total 110.11 last 88.34)
total: 10000 open/close in 113.10 seconds: 88.42 ops/second
 - unlinked 0 (time 1787600488 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 884 (time 1787600575.02 total 10.01 last 88.34)
 - open/close 1813 (time 1787600585.03 total 20.02 last 92.80)
 - open/close 2670 (time 1787600595.03 total 30.02 last 85.63)
 - open/close 3498 (time 1787600605.04 total 40.03 last 82.72)
 - open/close 4436 (time 1787600615.05 total 50.04 last 93.78)
 - open/close 5278 (time 1787600625.06 total 60.05 last 84.11)
 - open/close 6204 (time 1787600635.06 total 70.06 last 92.53)
 - open/close 7144 (time 1787600645.07 total 80.06 last 93.98)
 - open/close 8024 (time 1787600655.07 total 90.06 last 87.97)
 - open/close 8805 (time 1787600665.07 total 100.06 last 78.08)
 - open/close 9679 (time 1787600675.08 total 110.07 last 87.32)
total: 10000 open/close in 114.22 seconds: 87.55 ops/second
 - unlinked 0 (time 1787600683 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 937 (time 1787600771.76 total 10.00 last 93.69)
 - open/close 1753 (time 1787600781.76 total 20.00 last 81.58)
 - open/close 2595 (time 1787600791.77 total 30.01 last 84.13)
 - open/close 3353 (time 1787600801.77 total 40.02 last 75.77)
 - open/close 4183 (time 1787600811.78 total 50.02 last 82.96)
 - open/close 5009 (time 1787600821.78 total 60.02 last 82.59)
 - open/close 5739 (time 1787600831.80 total 70.04 last 72.89)
 - open/close 6493 (time 1787600841.80 total 80.04 last 75.35)
 - open/close 7263 (time 1787600851.81 total 90.05 last 76.92)
 - open/close 7997 (time 1787600861.82 total 100.06 last 73.34)
 - open/close 8670 (time 1787600871.83 total 110.07 last 67.22)
 - open/close 9325 (time 1787600881.83 total 120.08 last 65.49)
total: 10000 open/close in 130.02 seconds: 76.91 ops/second
 - unlinked 0 (time 1787600898 ; total 1 ; last 1)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 856 (time 1787600987.42 total 10.01 last 85.48)
 - open/close 1603 (time 1787600997.48 total 20.07 last 74.31)
 - open/close 2401 (time 1787601007.48 total 30.07 last 79.78)
 - open/close 3142 (time 1787601017.50 total 40.09 last 73.92)
 - open/close 3919 (time 1787601027.51 total 50.10 last 77.64)
 - open/close 4744 (time 1787601037.51 total 60.10 last 82.50)
 - open/close 5498 (time 1787601047.53 total 70.12 last 75.28)
 - open/close 6374 (time 1787601057.54 total 80.13 last 87.51)
 - open/close 7190 (time 1787601067.54 total 90.13 last 81.56)
 - open/close 7972 (time 1787601077.56 total 100.15 last 78.07)
 - open/close 8755 (time 1787601087.57 total 110.16 last 78.21)
 - open/close 9593 (time 1787601097.57 total 120.16 last 83.78)
total: 10000 open/close in 125.71 seconds: 79.55 ops/second
 - unlinked 0 (time 1787601107 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 717 (time 1787601206.42 total 10.00 last 71.67)
 - open/close 1524 (time 1787601216.43 total 20.01 last 80.64)
 - open/close 2352 (time 1787601226.44 total 30.02 last 82.74)
 - open/close 3145 (time 1787601236.44 total 40.03 last 79.24)
 - open/close 3977 (time 1787601246.45 total 50.03 last 83.17)
 - open/close 4831 (time 1787601256.45 total 60.03 last 85.39)
 - open/close 5658 (time 1787601266.46 total 70.04 last 82.60)
 - open/close 6445 (time 1787601276.47 total 80.05 last 78.62)
 - open/close 7233 (time 1787601286.47 total 90.06 last 78.78)
 - open/close 7918 (time 1787601296.48 total 100.06 last 68.47)
 - open/close 8705 (time 1787601306.49 total 110.07 last 78.60)
 - open/close 9540 (time 1787601316.50 total 120.08 last 83.42)
total: 10000 open/close in 125.55 seconds: 79.65 ops/second
 - unlinked 0 (time 1787601325 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 879 (time 1787601413.57 total 10.01 last 87.84)
 - open/close 1772 (time 1787601423.58 total 20.01 last 89.26)
 - open/close 2664 (time 1787601433.58 total 30.02 last 89.16)
 - open/close 3553 (time 1787601443.58 total 40.02 last 88.89)
 - open/close 4358 (time 1787601453.59 total 50.02 last 80.44)
 - open/close 5124 (time 1787601463.59 total 60.02 last 76.60)
 - open/close 6029 (time 1787601473.59 total 70.03 last 90.45)
 - open/close 6880 (time 1787601483.60 total 80.03 last 85.07)
 - open/close 7760 (time 1787601493.60 total 90.04 last 87.96)
 - open/close 8698 (time 1787601503.61 total 100.04 last 93.73)
 - open/close 9580 (time 1787601513.61 total 110.05 last 88.16)
total: 10000 open/close in 114.78 seconds: 87.12 ops/second
 - unlinked 0 (time 1787601522 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg625-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg625-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg625-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1
Started lustre-OST0000
oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg625-server: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg625-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg625-client.virtnet:  -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre
On OST0, 14326 used inodes rc=0
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg625-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg625-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.125@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.125@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.206.125@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg625-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1
Started lustre-OST0000
oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg625-server: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg625-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg625-client.virtnet:  -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg625-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg625-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server
stop mds service on oleg625-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server

unloading modules via unload_modules_local on: 'oleg625-server'
oleg625-server: oleg625-server.virtnet: executing unload_modules_local
oleg625-server: 
modules unloaded.
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2
pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2
pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2
pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2