== conf-sanity test 69: replace an OST with the same index ========================================================== 10:50:02 (1782485402)
start mds service on oleg438-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg438-server'
oleg438-server: oleg438-server.virtnet: executing load_modules_local
oleg438-server: Loading modules from /home/green/git/lustre-release/lustre
oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg438-server: detected 4 online CPUs by sysfs
oleg438-server: MODOPTS_LIBCFS=
oleg438-server: Force libcfs to create 2 CPU partitions
oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg438-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg438-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1
Started lustre-OST0000
oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg438-client.virtnet:  -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1102 (time 1782485467.55 total 10.00 last 110.15)
 - open/close 2103 (time 1782485477.56 total 20.01 last 100.06)
 - open/close 3088 (time 1782485487.56 total 30.01 last 98.47)
 - open/close 4126 (time 1782485497.57 total 40.02 last 103.72)
 - open/close 5239 (time 1782485507.57 total 50.02 last 111.26)
 - open/close 6328 (time 1782485517.58 total 60.03 last 108.83)
 - open/close 7371 (time 1782485527.58 total 70.03 last 104.27)
 - open/close 8762 (time 1782485537.58 total 80.04 last 139.07)
 - open/close 9840 (time 1782485547.59 total 90.04 last 107.78)
total: 10000 open/close in 91.72 seconds: 109.03 ops/second
 - unlinked 0 (time 1782485551 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 1344 (time 1782485621.56 total 10.01 last 134.31)
 - open/close 2862 (time 1782485631.56 total 20.01 last 151.79)
 - open/close 3954 (time 1782485641.56 total 30.02 last 109.12)
 - open/close 4928 (time 1782485651.56 total 40.02 last 97.40)
 - open/close 6064 (time 1782485661.57 total 50.02 last 113.51)
 - open/close 7174 (time 1782485671.59 total 60.04 last 110.83)
 - open/close 8178 (time 1782485681.60 total 70.05 last 100.31)
 - open/close 8932 (time 1782485691.60 total 80.05 last 75.39)
 - open/close 9422 (time 1782485701.60 total 90.05 last 49.00)
 - open/close 9844 (time 1782485711.73 total 100.18 last 41.64)
total: 10000 open/close in 105.97 seconds: 94.36 ops/second
 - unlinked 0 (time 1782485724 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 711 (time 1782485839.53 total 10.01 last 71.05)
 - open/close 1402 (time 1782485849.54 total 20.02 last 69.04)
 - open/close 2061 (time 1782485859.56 total 30.04 last 65.76)
 - open/close 2739 (time 1782485869.57 total 40.05 last 67.72)
 - open/close 3399 (time 1782485879.57 total 50.05 last 65.98)
 - open/close 4072 (time 1782485889.59 total 60.06 last 67.22)
 - open/close 4695 (time 1782485899.60 total 70.08 last 62.21)
 - open/close 5316 (time 1782485909.62 total 80.10 last 62.00)
 - open/close 5902 (time 1782485919.64 total 90.11 last 58.49)
 - open/close 6460 (time 1782485929.64 total 100.12 last 55.75)
 - open/close 7074 (time 1782485939.65 total 110.13 last 61.36)
 - open/close 7648 (time 1782485949.66 total 120.14 last 57.35)
 - open/close 8231 (time 1782485959.67 total 130.15 last 58.25)
 - open/close 8784 (time 1782485969.67 total 140.15 last 55.28)
 - open/close 9369 (time 1782485979.68 total 150.16 last 58.43)
 - open/close 9926 (time 1782485989.70 total 160.17 last 55.63)
total: 10000 open/close in 161.30 seconds: 62.00 ops/second
 - unlinked 0 (time 1782485995 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 702 (time 1782486109.78 total 10.00 last 70.18)
 - open/close 1447 (time 1782486119.79 total 20.02 last 74.41)
 - open/close 2228 (time 1782486129.81 total 30.03 last 77.99)
 - open/close 3043 (time 1782486139.82 total 40.04 last 81.44)
 - open/close 3906 (time 1782486149.82 total 50.04 last 86.30)
 - open/close 4864 (time 1782486159.82 total 60.04 last 95.79)
 - open/close 5796 (time 1782486169.82 total 70.04 last 93.16)
 - open/close 6719 (time 1782486179.82 total 80.04 last 92.29)
 - open/close 7717 (time 1782486189.82 total 90.04 last 99.79)
 - open/close 8648 (time 1782486199.83 total 100.05 last 93.08)
 - open/close 9679 (time 1782486209.83 total 110.05 last 103.06)
total: 10000 open/close in 112.63 seconds: 88.79 ops/second
 - unlinked 0 (time 1782486214 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 1211 (time 1782486284.74 total 10.00 last 121.05)
 - open/close 2258 (time 1782486294.75 total 20.01 last 104.62)
 - open/close 3179 (time 1782486304.76 total 30.02 last 92.03)
 - open/close 4186 (time 1782486314.76 total 40.02 last 100.69)
 - open/close 5221 (time 1782486324.77 total 50.03 last 103.44)
 - open/close 6224 (time 1782486334.77 total 60.03 last 100.24)
 - open/close 7199 (time 1782486344.78 total 70.04 last 97.46)
 - open/close 8254 (time 1782486354.78 total 80.04 last 105.48)
 - open/close 9364 (time 1782486364.78 total 90.04 last 110.96)
total: 10000 open/close in 96.77 seconds: 103.34 ops/second
 - unlinked 0 (time 1782486374 ; total 0 ; last 0)
total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second
 - open/close 1134 (time 1782486440.85 total 10.01 last 113.33)
 - open/close 2264 (time 1782486450.85 total 20.01 last 112.95)
 - open/close 3441 (time 1782486460.86 total 30.02 last 117.65)
 - open/close 4610 (time 1782486470.86 total 40.02 last 116.83)
 - open/close 5800 (time 1782486480.87 total 50.02 last 118.96)
 - open/close 6830 (time 1782486490.87 total 60.03 last 102.95)
 - open/close 7959 (time 1782486500.88 total 70.04 last 112.81)
 - open/close 9194 (time 1782486510.88 total 80.04 last 123.45)
total: 10000 open/close in 87.41 seconds: 114.41 ops/second
 - unlinked 0 (time 1782486520 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1342 (time 1782486577.32 total 10.01 last 134.05)
 - open/close 2786 (time 1782486587.32 total 20.01 last 144.38)
 - open/close 4172 (time 1782486597.33 total 30.02 last 138.49)
 - open/close 5435 (time 1782486607.34 total 40.03 last 126.16)
 - open/close 6743 (time 1782486617.34 total 50.04 last 130.72)
 - open/close 7922 (time 1782486627.35 total 60.04 last 117.85)
 - open/close 9333 (time 1782486637.35 total 70.04 last 141.09)
total: 10000 open/close in 75.61 seconds: 132.25 ops/second
 - unlinked 0 (time 1782486645 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 816 (time 1782486759.28 total 10.01 last 81.53)
 - open/close 1608 (time 1782486769.28 total 20.01 last 79.17)
 - open/close 2485 (time 1782486779.29 total 30.02 last 87.66)
 - open/close 3408 (time 1782486789.31 total 40.04 last 92.11)
 - open/close 4237 (time 1782486799.31 total 50.04 last 82.87)
 - open/close 5047 (time 1782486809.31 total 60.04 last 80.99)
 - open/close 5884 (time 1782486819.31 total 70.04 last 83.69)
 - open/close 6710 (time 1782486829.32 total 80.05 last 82.53)
 - open/close 7559 (time 1782486839.33 total 90.06 last 84.85)
 - open/close 8397 (time 1782486849.34 total 100.07 last 83.70)
 - open/close 9256 (time 1782486859.34 total 110.07 last 85.89)
total: 10000 open/close in 119.06 seconds: 83.99 ops/second
 - unlinked 0 (time 1782486871 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 747 (time 1782486963.02 total 10.01 last 74.62)
 - open/close 1480 (time 1782486973.03 total 20.02 last 73.21)
 - open/close 2274 (time 1782486983.03 total 30.03 last 79.36)
 - open/close 3086 (time 1782486993.04 total 40.04 last 81.13)
 - open/close 3876 (time 1782487003.05 total 50.04 last 78.95)
 - open/close 4649 (time 1782487013.05 total 60.05 last 77.26)
 - open/close 5458 (time 1782487023.06 total 70.05 last 80.87)
 - open/close 6312 (time 1782487033.06 total 80.06 last 85.37)
 - open/close 7089 (time 1782487043.07 total 90.06 last 77.63)
 - open/close 7953 (time 1782487053.08 total 100.07 last 86.32)
 - open/close 8754 (time 1782487063.08 total 110.08 last 80.08)
 - open/close 9581 (time 1782487073.09 total 120.09 last 82.62)
total: 10000 open/close in 125.12 seconds: 79.92 ops/second
 - unlinked 0 (time 1782487080 ; total 0 ; last 0)
total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second
 - open/close 1356 (time 1782487144.57 total 10.00 last 135.55)
 - open/close 3019 (time 1782487154.57 total 20.01 last 166.26)
 - open/close 4710 (time 1782487164.58 total 30.01 last 169.06)
 - open/close 6067 (time 1782487174.58 total 40.01 last 135.70)
 - open/close 7721 (time 1782487184.58 total 50.01 last 165.36)
 - open/close 9264 (time 1782487194.58 total 60.02 last 154.23)
total: 10000 open/close in 65.62 seconds: 152.39 ops/second
 - unlinked 0 (time 1782487202 ; total 0 ; last 0)
total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg438-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg438-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg438-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1
Started lustre-OST0000
oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg438-client.virtnet:  -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre
On OST0, 6318 used inodes rc=0
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg438-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg438-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.138@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.138@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.204.138@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg438-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1
Started lustre-OST0000
oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg438-client.virtnet:  -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg438-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg438-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server
stop mds service on oleg438-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server

unloading modules via unload_modules_local on: 'oleg438-server'
oleg438-server: oleg438-server.virtnet: executing unload_modules_local
oleg438-server: 
modules unloaded.
pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2
pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2
pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2