== conf-sanity test 69: replace an OST with the same index ========================================================== 12:33:11 (1774888391)
start mds service on oleg343-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg343-server'
oleg343-server: oleg343-server.virtnet: executing load_modules_local
oleg343-server: Loading modules from /home/green/git/lustre-release/lustre
oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg343-server: detected 4 online CPUs by sysfs
oleg343-server: MODOPTS_LIBCFS=
oleg343-server: Force libcfs to create 2 CPU partitions
oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg343-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-OST0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg343-client.virtnet:  -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 737 (time 1774888461.28 total 10.00 last 73.70)
 - open/close 1455 (time 1774888471.30 total 20.01 last 71.70)
 - open/close 2212 (time 1774888481.30 total 30.02 last 75.64)
 - open/close 2921 (time 1774888491.31 total 40.02 last 70.89)
 - open/close 3580 (time 1774888501.31 total 50.03 last 65.87)
 - open/close 4325 (time 1774888511.32 total 60.03 last 74.45)
 - open/close 4967 (time 1774888521.33 total 70.04 last 64.14)
 - open/close 5663 (time 1774888531.33 total 80.05 last 69.57)
 - open/close 6251 (time 1774888541.34 total 90.06 last 58.75)
 - open/close 6871 (time 1774888551.35 total 100.06 last 61.95)
 - open/close 7454 (time 1774888561.35 total 110.07 last 58.29)
 - open/close 8111 (time 1774888571.36 total 120.08 last 65.63)
 - open/close 8629 (time 1774888581.37 total 130.08 last 51.76)
 - open/close 9175 (time 1774888591.38 total 140.09 last 54.54)
 - open/close 9722 (time 1774888601.39 total 150.11 last 54.61)
total: 10000 open/close in 155.24 seconds: 64.42 ops/second
 - unlinked 0 (time 1774888611 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 714 (time 1774888710.69 total 10.00 last 71.37)
 - open/close 1441 (time 1774888720.70 total 20.01 last 72.68)
 - open/close 2229 (time 1774888730.72 total 30.03 last 78.62)
 - open/close 3030 (time 1774888740.72 total 40.03 last 80.05)
 - open/close 3856 (time 1774888750.73 total 50.04 last 82.58)
 - open/close 4574 (time 1774888760.73 total 60.04 last 71.77)
 - open/close 5289 (time 1774888770.75 total 70.06 last 71.34)
 - open/close 6028 (time 1774888780.76 total 80.07 last 73.84)
 - open/close 6820 (time 1774888790.77 total 90.08 last 79.17)
 - open/close 7580 (time 1774888800.77 total 100.08 last 75.95)
 - open/close 8401 (time 1774888810.78 total 110.09 last 82.06)
 - open/close 9216 (time 1774888820.78 total 120.09 last 81.49)
 - open/close 9953 (time 1774888830.79 total 130.10 last 73.62)
total: 10000 open/close in 130.78 seconds: 76.47 ops/second
 - unlinked 0 (time 1774888835 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 744 (time 1774888928.44 total 10.01 last 74.32)
 - open/close 1532 (time 1774888938.45 total 20.02 last 78.76)
 - open/close 2310 (time 1774888948.45 total 30.02 last 77.77)
 - open/close 3075 (time 1774888958.46 total 40.03 last 76.43)
 - open/close 3853 (time 1774888968.46 total 50.03 last 77.77)
 - open/close 4640 (time 1774888978.47 total 60.04 last 78.64)
 - open/close 5443 (time 1774888988.48 total 70.05 last 80.23)
 - open/close 6212 (time 1774888998.50 total 80.07 last 76.72)
 - open/close 7024 (time 1774889008.51 total 90.07 last 81.18)
 - open/close 7799 (time 1774889018.51 total 100.08 last 77.48)
 - open/close 8577 (time 1774889028.51 total 110.08 last 77.76)
 - open/close 9362 (time 1774889038.52 total 120.09 last 78.45)
total: 10000 open/close in 128.44 seconds: 77.86 ops/second
 - unlinked 0 (time 1774889050 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 763 (time 1774889141.21 total 10.01 last 76.22)
 - open/close 1565 (time 1774889151.22 total 20.02 last 80.14)
 - open/close 2325 (time 1774889161.22 total 30.02 last 75.95)
 - open/close 3119 (time 1774889171.23 total 40.03 last 79.36)
 - open/close 3942 (time 1774889181.23 total 50.03 last 82.30)
 - open/close 4751 (time 1774889191.25 total 60.05 last 80.70)
 - open/close 5527 (time 1774889201.25 total 70.06 last 77.60)
 - open/close 6271 (time 1774889211.26 total 80.06 last 74.37)
 - open/close 7055 (time 1774889221.27 total 90.07 last 78.34)
 - open/close 7876 (time 1774889231.27 total 100.08 last 82.04)
 - open/close 8625 (time 1774889241.28 total 110.08 last 74.85)
 - open/close 9442 (time 1774889251.29 total 120.09 last 81.62)
total: 10000 open/close in 127.19 seconds: 78.62 ops/second
 - unlinked 0 (time 1774889261 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 794 (time 1774889347.80 total 10.02 last 79.23)
 - open/close 1558 (time 1774889357.81 total 20.03 last 76.36)
 - open/close 2382 (time 1774889367.81 total 30.03 last 82.39)
 - open/close 3194 (time 1774889377.82 total 40.04 last 81.10)
 - open/close 4013 (time 1774889387.82 total 50.04 last 81.86)
 - open/close 4773 (time 1774889397.84 total 60.06 last 75.86)
 - open/close 5634 (time 1774889407.85 total 70.07 last 86.07)
 - open/close 6452 (time 1774889417.85 total 80.07 last 81.79)
 - open/close 7340 (time 1774889427.86 total 90.08 last 88.68)
 - open/close 8181 (time 1774889437.87 total 100.09 last 84.03)
 - open/close 9068 (time 1774889447.87 total 110.09 last 88.66)
 - open/close 9955 (time 1774889457.89 total 120.11 last 88.58)
total: 10000 open/close in 120.64 seconds: 82.89 ops/second
 - unlinked 0 (time 1774889461 ; total 0 ; last 0)
total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second
 - open/close 930 (time 1774889535.20 total 10.01 last 92.94)
 - open/close 1822 (time 1774889545.21 total 20.01 last 89.16)
 - open/close 2737 (time 1774889555.22 total 30.02 last 91.38)
 - open/close 3605 (time 1774889565.24 total 40.05 last 86.60)
 - open/close 4497 (time 1774889575.25 total 50.06 last 89.11)
 - open/close 5373 (time 1774889585.26 total 60.07 last 87.49)
 - open/close 6315 (time 1774889595.26 total 70.07 last 94.18)
 - open/close 7214 (time 1774889605.27 total 80.07 last 89.87)
 - open/close 8136 (time 1774889615.28 total 90.08 last 92.11)
 - open/close 8929 (time 1774889625.28 total 100.08 last 79.30)
 - open/close 9848 (time 1774889635.28 total 110.09 last 91.84)
total: 10000 open/close in 111.72 seconds: 89.51 ops/second
 - unlinked 0 (time 1774889639 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 882 (time 1774889714.76 total 10.00 last 88.16)
 - open/close 1797 (time 1774889724.77 total 20.01 last 91.45)
 - open/close 2656 (time 1774889734.77 total 30.02 last 85.85)
 - open/close 3511 (time 1774889744.77 total 40.02 last 85.49)
 - open/close 4317 (time 1774889754.78 total 50.03 last 80.54)
 - open/close 5142 (time 1774889764.79 total 60.03 last 82.45)
 - open/close 6097 (time 1774889774.79 total 70.04 last 95.46)
 - open/close 7021 (time 1774889784.80 total 80.04 last 92.31)
 - open/close 8046 (time 1774889794.81 total 90.05 last 102.43)
 - open/close 8970 (time 1774889804.82 total 100.06 last 92.32)
total: 10000 open/close in 109.92 seconds: 90.97 ops/second
 - unlinked 0 (time 1774889816 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 889 (time 1774889887.78 total 10.01 last 88.85)
 - open/close 1691 (time 1774889897.79 total 20.01 last 80.14)
 - open/close 2604 (time 1774889907.80 total 30.02 last 91.20)
 - open/close 3463 (time 1774889917.81 total 40.03 last 85.88)
 - open/close 4488 (time 1774889927.81 total 50.03 last 102.49)
 - open/close 5412 (time 1774889937.81 total 60.03 last 92.40)
 - open/close 6290 (time 1774889947.81 total 70.03 last 87.75)
 - open/close 7142 (time 1774889957.82 total 80.04 last 85.17)
 - open/close 7957 (time 1774889967.82 total 90.04 last 81.49)
 - open/close 8793 (time 1774889977.82 total 100.04 last 83.56)
 - open/close 9666 (time 1774889987.82 total 110.05 last 87.29)
total: 10000 open/close in 113.83 seconds: 87.85 ops/second
 - unlinked 0 (time 1774889994 ; total 0 ; last 0)
total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second
 - open/close 959 (time 1774890061.98 total 10.01 last 95.82)
 - open/close 2048 (time 1774890071.98 total 20.01 last 108.88)
 - open/close 3082 (time 1774890081.99 total 30.02 last 103.31)
 - open/close 4256 (time 1774890092.01 total 40.04 last 117.17)
 - open/close 5275 (time 1774890102.01 total 50.04 last 101.85)
 - open/close 6297 (time 1774890112.01 total 60.04 last 102.19)
 - open/close 7404 (time 1774890122.02 total 70.05 last 110.65)
 - open/close 8440 (time 1774890132.02 total 80.06 last 103.53)
 - open/close 9416 (time 1774890142.03 total 90.06 last 97.54)
total: 10000 open/close in 95.69 seconds: 104.50 ops/second
 - unlinked 0 (time 1774890150 ; total 0 ; last 0)
total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second
 - open/close 1059 (time 1774890220.59 total 10.00 last 105.89)
 - open/close 2198 (time 1774890230.59 total 20.01 last 113.86)
 - open/close 3275 (time 1774890240.60 total 30.01 last 107.61)
 - open/close 4396 (time 1774890250.60 total 40.01 last 112.10)
 - open/close 5456 (time 1774890260.60 total 50.01 last 105.99)
 - open/close 6547 (time 1774890270.60 total 60.02 last 109.07)
 - open/close 7631 (time 1774890280.61 total 70.02 last 108.33)
 - open/close 8842 (time 1774890290.62 total 80.04 last 120.96)
total: 10000 open/close in 89.47 seconds: 111.77 ops/second
 - unlinked 0 (time 1774890302 ; total 0 ; last 0)
total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg343-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg343-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.143@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.143@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg343-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-OST0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg343-server: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg343-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg343-client.virtnet:  -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg343-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg343-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.143@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.143@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.143@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.143@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.143@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg343-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-OST0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg343-server: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg343-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg343-client.virtnet:  -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg343-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg343-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server
stop mds service on oleg343-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server

unloading modules via unload_modules_local on: 'oleg343-server'
oleg343-server: oleg343-server.virtnet: executing unload_modules_local
oleg343-server: 
modules unloaded.
pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2
pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2