== conf-sanity test 69: replace an OST with the same index ========================================================== 08:29:37 (1788438577)
start mds service on oleg459-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing load_modules_local
oleg459-server: kmemleak disabled
oleg459-server: Loading modules from /home/green/git/lustre-release/lustre
oleg459-server: detected 4 online CPUs by sysfs
oleg459-server: MODOPTS_LIBCFS=
oleg459-server: Force libcfs to create 2 CPU partitions
oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg459-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg459-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 674 (time 1788438654.83 total 10.01 last 67.35)
 - open/close 1333 (time 1788438664.84 total 20.02 last 65.81)
 - open/close 2039 (time 1788438674.84 total 30.02 last 70.59)
 - open/close 2679 (time 1788438684.85 total 40.03 last 63.96)
 - open/close 3348 (time 1788438694.86 total 50.04 last 66.84)
 - open/close 4084 (time 1788438704.88 total 60.06 last 73.43)
 - open/close 4814 (time 1788438714.88 total 70.06 last 72.98)
 - open/close 5470 (time 1788438724.89 total 80.07 last 65.55)
 - open/close 6139 (time 1788438734.89 total 90.08 last 66.87)
 - open/close 6750 (time 1788438744.90 total 100.08 last 61.08)
 - open/close 7448 (time 1788438754.90 total 110.08 last 69.77)
 - open/close 8182 (time 1788438764.90 total 120.09 last 73.37)
 - open/close 8843 (time 1788438774.92 total 130.10 last 65.99)
 - open/close 9554 (time 1788438784.93 total 140.12 last 71.01)
total: 10000 open/close in 146.72 seconds: 68.16 ops/second
 - unlinked 0 (time 1788438796 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 613 (time 1788438903.72 total 10.01 last 61.24)
 - open/close 1185 (time 1788438913.73 total 20.01 last 57.18)
 - open/close 1859 (time 1788438923.73 total 30.02 last 67.38)
 - open/close 2474 (time 1788438933.73 total 40.02 last 61.50)
 - open/close 3171 (time 1788438943.73 total 50.02 last 69.70)
 - open/close 3817 (time 1788438953.73 total 60.02 last 64.59)
 - open/close 4411 (time 1788438963.73 total 70.02 last 59.38)
 - open/close 5175 (time 1788438973.74 total 80.03 last 76.35)
 - open/close 5878 (time 1788438983.74 total 90.03 last 70.28)
 - open/close 6525 (time 1788438993.75 total 100.04 last 64.66)
 - open/close 7243 (time 1788439003.75 total 110.04 last 71.79)
 - open/close 7870 (time 1788439013.75 total 120.04 last 62.70)
 - open/close 8532 (time 1788439023.76 total 130.05 last 66.16)
 - open/close 9214 (time 1788439033.76 total 140.05 last 68.18)
 - open/close 9874 (time 1788439043.77 total 150.05 last 65.96)
total: 10000 open/close in 152.01 seconds: 65.78 ops/second
 - unlinked 0 (time 1788439050 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 733 (time 1788439158.19 total 10.00 last 73.27)
 - open/close 1414 (time 1788439168.20 total 20.01 last 68.04)
 - open/close 2019 (time 1788439178.20 total 30.02 last 60.48)
 - open/close 2675 (time 1788439188.22 total 40.03 last 65.53)
 - open/close 3302 (time 1788439198.23 total 50.04 last 62.60)
 - open/close 3942 (time 1788439208.24 total 60.05 last 63.95)
 - open/close 4541 (time 1788439218.24 total 70.05 last 59.90)
 - open/close 5231 (time 1788439228.25 total 80.06 last 68.94)
 - open/close 5851 (time 1788439238.25 total 90.06 last 61.98)
 - open/close 6461 (time 1788439248.26 total 100.07 last 60.95)
 - open/close 7113 (time 1788439258.27 total 110.08 last 65.15)
 - open/close 7786 (time 1788439268.28 total 120.09 last 67.25)
 - open/close 8479 (time 1788439278.30 total 130.11 last 69.17)
 - open/close 9131 (time 1788439288.30 total 140.11 last 65.20)
 - open/close 9778 (time 1788439298.30 total 150.11 last 64.67)
total: 10000 open/close in 153.32 seconds: 65.22 ops/second
 - unlinked 0 (time 1788439306 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 584 (time 1788439426.79 total 10.01 last 58.32)
 - open/close 1158 (time 1788439436.80 total 20.02 last 57.38)
 - open/close 1771 (time 1788439446.80 total 30.02 last 61.26)
 - open/close 2299 (time 1788439456.82 total 40.04 last 52.69)
 - open/close 2800 (time 1788439466.85 total 50.07 last 49.95)
 - open/close 3266 (time 1788439476.86 total 60.08 last 46.59)
 - open/close 3686 (time 1788439486.86 total 70.08 last 41.99)
 - open/close 4139 (time 1788439496.86 total 80.08 last 45.27)
 - open/close 4708 (time 1788439506.89 total 90.10 last 56.78)
 - open/close 5397 (time 1788439516.89 total 100.11 last 68.89)
 - open/close 5992 (time 1788439526.90 total 110.11 last 59.45)
 - open/close 6627 (time 1788439536.91 total 120.13 last 63.39)
 - open/close 7259 (time 1788439546.92 total 130.14 last 63.14)
 - open/close 7812 (time 1788439556.94 total 140.16 last 55.20)
 - open/close 8339 (time 1788439566.95 total 150.17 last 52.65)
 - open/close 8875 (time 1788439576.96 total 160.18 last 53.54)
 - open/close 9355 (time 1788439586.97 total 170.19 last 47.95)
 - open/close 9957 (time 1788439596.99 total 180.21 last 60.11)
total: 10000 open/close in 180.96 seconds: 55.26 ops/second
 - unlinked 0 (time 1788439601 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 596 (time 1788439721.00 total 10.00 last 59.57)
 - open/close 1237 (time 1788439731.00 total 20.00 last 64.10)
 - open/close 1863 (time 1788439741.04 total 30.04 last 62.38)
 - open/close 2518 (time 1788439751.04 total 40.05 last 65.45)
 - open/close 3142 (time 1788439761.05 total 50.06 last 62.33)
 - open/close 3750 (time 1788439771.06 total 60.07 last 60.75)
 - open/close 4407 (time 1788439781.07 total 70.08 last 65.63)
 - open/close 5065 (time 1788439791.08 total 80.08 last 65.75)
 - open/close 5665 (time 1788439801.09 total 90.09 last 59.95)
 - open/close 6328 (time 1788439811.10 total 100.10 last 66.25)
 - open/close 6878 (time 1788439821.11 total 110.11 last 54.94)
 - open/close 7441 (time 1788439831.13 total 120.13 last 56.19)
 - open/close 7959 (time 1788439841.13 total 130.14 last 51.79)
 - open/close 8504 (time 1788439851.14 total 140.14 last 54.45)
 - open/close 9069 (time 1788439861.15 total 150.16 last 56.43)
 - open/close 9584 (time 1788439871.16 total 160.17 last 51.45)
total: 10000 open/close in 168.69 seconds: 59.28 ops/second
 - unlinked 0 (time 1788439885 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 666 (time 1788440008.56 total 10.01 last 66.54)
 - open/close 1296 (time 1788440018.56 total 20.01 last 62.99)
 - open/close 1952 (time 1788440028.57 total 30.02 last 65.56)
 - open/close 2604 (time 1788440038.57 total 40.02 last 65.18)
 - open/close 3203 (time 1788440048.58 total 50.03 last 59.87)
 - open/close 3801 (time 1788440058.59 total 60.04 last 59.70)
 - open/close 4391 (time 1788440068.61 total 70.06 last 58.91)
 - open/close 4941 (time 1788440078.61 total 80.06 last 54.99)
 - open/close 5554 (time 1788440088.61 total 90.06 last 61.30)
 - open/close 6163 (time 1788440098.63 total 100.08 last 60.78)
 - open/close 6797 (time 1788440108.63 total 110.08 last 63.39)
 - open/close 7367 (time 1788440118.64 total 120.09 last 56.96)
 - open/close 7885 (time 1788440128.64 total 130.09 last 51.80)
 - open/close 8511 (time 1788440138.64 total 140.09 last 62.59)
 - open/close 9049 (time 1788440148.65 total 150.10 last 53.76)
 - open/close 9661 (time 1788440158.65 total 160.10 last 61.17)
total: 10000 open/close in 164.86 seconds: 60.66 ops/second
 - unlinked 0 (time 1788440168 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 582 (time 1788440286.95 total 10.01 last 58.14)
 - open/close 1224 (time 1788440296.96 total 20.01 last 64.17)
 - open/close 1810 (time 1788440306.96 total 30.02 last 58.57)
 - open/close 2405 (time 1788440316.97 total 40.02 last 59.48)
 - open/close 2982 (time 1788440327.02 total 50.08 last 57.37)
 - open/close 3544 (time 1788440337.03 total 60.09 last 56.17)
 - open/close 4204 (time 1788440347.03 total 70.09 last 65.99)
 - open/close 4829 (time 1788440357.04 total 80.10 last 62.41)
 - open/close 5393 (time 1788440367.06 total 90.12 last 56.31)
 - open/close 5899 (time 1788440377.07 total 100.12 last 50.58)
 - open/close 6367 (time 1788440387.07 total 110.13 last 46.78)
 - open/close 6848 (time 1788440397.08 total 120.13 last 48.07)
 - open/close 7462 (time 1788440407.08 total 130.14 last 61.35)
 - open/close 8063 (time 1788440417.09 total 140.14 last 60.09)
 - open/close 8599 (time 1788440427.10 total 150.15 last 53.54)
 - open/close 9162 (time 1788440437.10 total 160.16 last 56.28)
 - open/close 9754 (time 1788440447.13 total 170.19 last 59.01)
total: 10000 open/close in 174.47 seconds: 57.32 ops/second
 - unlinked 0 (time 1788440455 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 495 (time 1788440578.08 total 10.00 last 49.49)
 - open/close 1048 (time 1788440588.12 total 20.03 last 55.13)
 - open/close 1535 (time 1788440598.12 total 30.04 last 48.67)
 - open/close 2139 (time 1788440608.13 total 40.05 last 60.34)
 - open/close 2735 (time 1788440618.14 total 50.06 last 59.54)
 - open/close 3314 (time 1788440628.15 total 60.07 last 57.83)
 - open/close 3859 (time 1788440638.15 total 70.07 last 54.50)
 - open/close 4436 (time 1788440648.16 total 80.07 last 57.68)
 - open/close 5073 (time 1788440658.18 total 90.09 last 63.58)
 - open/close 5666 (time 1788440668.19 total 100.11 last 59.19)
 - open/close 6258 (time 1788440678.23 total 110.15 last 58.99)
 - open/close 6855 (time 1788440688.24 total 120.15 last 59.65)
 - open/close 7486 (time 1788440698.24 total 130.16 last 63.07)
 - open/close 8102 (time 1788440708.24 total 140.16 last 61.58)
 - open/close 8674 (time 1788440718.25 total 150.17 last 57.17)
 - open/close 9253 (time 1788440728.26 total 160.18 last 57.84)
 - open/close 9726 (time 1788440738.28 total 170.20 last 47.21)
total: 10000 open/close in 175.25 seconds: 57.06 ops/second
 - unlinked 0 (time 1788440748 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 624 (time 1788440865.62 total 10.01 last 62.34)
 - open/close 1226 (time 1788440875.63 total 20.02 last 60.14)
 - open/close 1867 (time 1788440885.64 total 30.03 last 64.03)
 - open/close 2494 (time 1788440895.65 total 40.04 last 62.64)
 - open/close 3160 (time 1788440905.67 total 50.06 last 66.47)
 - open/close 3792 (time 1788440915.70 total 60.08 last 63.03)
 - open/close 4415 (time 1788440925.70 total 70.08 last 62.30)
 - open/close 5050 (time 1788440935.71 total 80.10 last 63.43)
 - open/close 5724 (time 1788440945.71 total 90.10 last 67.37)
 - open/close 6279 (time 1788440955.71 total 100.10 last 55.49)
 - open/close 6908 (time 1788440965.72 total 110.11 last 62.85)
 - open/close 7468 (time 1788440975.73 total 120.12 last 55.93)
 - open/close 7961 (time 1788440985.75 total 130.14 last 49.24)
 - open/close 8433 (time 1788440995.79 total 140.18 last 47.00)
 - open/close 8964 (time 1788441005.81 total 150.20 last 52.98)
 - open/close 9527 (time 1788441015.82 total 160.21 last 56.24)
total: 10000 open/close in 168.32 seconds: 59.41 ops/second
 - unlinked 0 (time 1788441028 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 607 (time 1788441142.59 total 10.00 last 60.69)
 - open/close 1198 (time 1788441152.60 total 20.01 last 59.07)
 - open/close 1822 (time 1788441162.60 total 30.01 last 62.36)
 - open/close 2422 (time 1788441172.61 total 40.02 last 59.95)
 - open/close 3024 (time 1788441182.66 total 50.07 last 59.95)
 - open/close 3666 (time 1788441192.66 total 60.07 last 64.20)
 - open/close 4279 (time 1788441202.66 total 70.07 last 61.30)
 - open/close 4847 (time 1788441212.69 total 80.10 last 56.58)
 - open/close 5347 (time 1788441222.71 total 90.12 last 49.92)
 - open/close 5930 (time 1788441232.75 total 100.16 last 58.07)
 - open/close 6547 (time 1788441242.76 total 110.17 last 61.66)
 - open/close 7143 (time 1788441252.77 total 120.18 last 59.53)
 - open/close 7687 (time 1788441262.77 total 130.18 last 54.38)
 - open/close 8153 (time 1788441272.78 total 140.19 last 46.56)
 - open/close 8718 (time 1788441282.78 total 150.19 last 56.50)
 - open/close 9245 (time 1788441292.79 total 160.20 last 52.64)
 - open/close 9814 (time 1788441302.82 total 170.23 last 56.72)
total: 10000 open/close in 173.44 seconds: 57.66 ops/second
 - unlinked 0 (time 1788441312 ; total 1 ; last 1)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server
stop mds service on oleg459-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server
unloading modules via unload_modules_local on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing unload_modules_local
oleg459-server: kmemleak disabled
modules unloaded.
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2