== conf-sanity test 69: replace an OST with the same index ========================================================== 03:06:30 (1789023990)
start mds service on oleg650-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931344 Sep  9 19:14 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg650-server'
oleg650-server: oleg650-server.virtnet: executing load_modules_local
oleg650-server: kmemleak disabled
oleg650-server: Loading modules from /home/green/git/lustre-release/lustre
oleg650-server: detected 4 online CPUs by sysfs
oleg650-server: MODOPTS_LIBCFS=
oleg650-server: Force libcfs to create 2 CPU partitions
oleg650-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg650-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg650-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg650-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg650-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1
Started lustre-OST0000
oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg650-client.virtnet:  -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 753 (time 1789024074.12 total 10.01 last 75.26)
 - open/close 1439 (time 1789024084.13 total 20.02 last 68.51)
 - open/close 2153 (time 1789024094.16 total 30.05 last 71.20)
 - open/close 2874 (time 1789024104.16 total 40.05 last 72.08)
 - open/close 3604 (time 1789024114.17 total 50.06 last 72.94)
 - open/close 4297 (time 1789024124.18 total 60.07 last 69.25)
 - open/close 5036 (time 1789024134.20 total 70.08 last 73.76)
 - open/close 5734 (time 1789024144.20 total 80.08 last 69.80)
 - open/close 6421 (time 1789024154.20 total 90.09 last 68.68)
 - open/close 7142 (time 1789024164.21 total 100.10 last 72.03)
 - open/close 7694 (time 1789024174.23 total 110.11 last 55.13)
 - open/close 8244 (time 1789024184.24 total 120.12 last 54.95)
 - open/close 8847 (time 1789024194.25 total 130.13 last 60.24)
 - open/close 9551 (time 1789024204.26 total 140.15 last 70.30)
total: 10000 open/close in 147.17 seconds: 67.95 ops/second
 - unlinked 0 (time 1789024215 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 745 (time 1789024319.59 total 10.01 last 74.45)
 - open/close 1544 (time 1789024329.59 total 20.01 last 79.84)
 - open/close 2282 (time 1789024339.61 total 30.03 last 73.69)
 - open/close 3022 (time 1789024349.61 total 40.03 last 73.98)
 - open/close 3680 (time 1789024359.62 total 50.04 last 65.75)
 - open/close 4403 (time 1789024369.63 total 60.05 last 72.20)
 - open/close 5111 (time 1789024379.64 total 70.06 last 70.72)
 - open/close 5805 (time 1789024389.66 total 80.08 last 69.30)
 - open/close 6548 (time 1789024399.67 total 90.09 last 74.21)
 - open/close 7233 (time 1789024409.68 total 100.10 last 68.45)
 - open/close 7903 (time 1789024419.68 total 110.10 last 66.97)
 - open/close 8560 (time 1789024429.69 total 120.11 last 65.66)
 - open/close 9141 (time 1789024439.69 total 130.11 last 58.09)
 - open/close 9841 (time 1789024449.70 total 140.12 last 69.96)
total: 10000 open/close in 141.97 seconds: 70.44 ops/second
 - unlinked 0 (time 1789024454 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 694 (time 1789024550.96 total 10.00 last 69.37)
 - open/close 1466 (time 1789024560.97 total 20.01 last 77.13)
 - open/close 2319 (time 1789024570.98 total 30.02 last 85.27)
 - open/close 3019 (time 1789024580.99 total 40.03 last 69.89)
 - open/close 3753 (time 1789024591.00 total 50.04 last 73.36)
 - open/close 4439 (time 1789024601.00 total 60.04 last 68.56)
 - open/close 5135 (time 1789024611.02 total 70.06 last 69.51)
 - open/close 5857 (time 1789024621.02 total 80.06 last 72.15)
 - open/close 6648 (time 1789024631.03 total 90.07 last 79.03)
 - open/close 7324 (time 1789024641.03 total 100.07 last 67.59)
 - open/close 8011 (time 1789024651.04 total 110.08 last 68.65)
 - open/close 8683 (time 1789024661.05 total 120.09 last 67.13)
 - open/close 9387 (time 1789024671.06 total 130.10 last 70.34)
total: 10000 open/close in 139.15 seconds: 71.87 ops/second
 - unlinked 0 (time 1789024683 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 760 (time 1789024786.16 total 10.01 last 75.94)
 - open/close 1378 (time 1789024796.18 total 20.03 last 61.66)
 - open/close 2153 (time 1789024806.19 total 30.04 last 77.42)
 - open/close 2877 (time 1789024816.20 total 40.05 last 72.30)
 - open/close 3656 (time 1789024826.21 total 50.06 last 77.83)
 - open/close 4397 (time 1789024836.21 total 60.06 last 74.10)
 - open/close 4995 (time 1789024846.22 total 70.08 last 59.73)
 - open/close 5728 (time 1789024856.24 total 80.09 last 73.22)
 - open/close 6433 (time 1789024866.24 total 90.09 last 70.47)
 - open/close 7115 (time 1789024876.24 total 100.10 last 68.17)
 - open/close 7742 (time 1789024886.26 total 110.11 last 62.61)
 - open/close 8485 (time 1789024896.26 total 120.11 last 74.30)
 - open/close 9225 (time 1789024906.29 total 130.14 last 73.80)
 - open/close 9905 (time 1789024916.29 total 140.15 last 67.94)
total: 10000 open/close in 141.30 seconds: 70.77 ops/second
 - unlinked 0 (time 1789024922 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 776 (time 1789025030.38 total 10.00 last 77.57)
 - open/close 1418 (time 1789025040.40 total 20.02 last 64.08)
 - open/close 2156 (time 1789025050.41 total 30.03 last 73.75)
 - open/close 2854 (time 1789025060.41 total 40.03 last 69.80)
 - open/close 3457 (time 1789025070.42 total 50.04 last 60.24)
 - open/close 4175 (time 1789025080.42 total 60.04 last 71.76)
 - open/close 4861 (time 1789025090.43 total 70.04 last 68.59)
 - open/close 5428 (time 1789025100.44 total 80.06 last 56.60)
 - open/close 6096 (time 1789025110.46 total 90.08 last 66.70)
 - open/close 6679 (time 1789025120.46 total 100.08 last 58.27)
 - open/close 7309 (time 1789025130.46 total 110.08 last 62.99)
 - open/close 7925 (time 1789025140.47 total 120.09 last 61.59)
 - open/close 8631 (time 1789025150.47 total 130.09 last 70.54)
 - open/close 9382 (time 1789025160.49 total 140.11 last 75.01)
total: 10000 open/close in 148.81 seconds: 67.20 ops/second
 - unlinked 0 (time 1789025173 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 708 (time 1789025276.40 total 10.02 last 70.68)
 - open/close 1397 (time 1789025286.40 total 20.02 last 68.89)
 - open/close 2091 (time 1789025296.41 total 30.02 last 69.37)
 - open/close 2802 (time 1789025306.41 total 40.03 last 71.07)
 - open/close 3479 (time 1789025316.42 total 50.04 last 67.61)
 - open/close 4158 (time 1789025326.43 total 60.05 last 67.86)
 - open/close 4778 (time 1789025336.43 total 70.05 last 61.99)
 - open/close 5419 (time 1789025346.43 total 80.05 last 64.09)
 - open/close 6127 (time 1789025356.45 total 90.07 last 70.68)
 - open/close 6888 (time 1789025366.46 total 100.08 last 76.02)
 - open/close 7474 (time 1789025376.48 total 110.09 last 58.50)
 - open/close 8018 (time 1789025386.51 total 120.13 last 54.21)
 - open/close 8654 (time 1789025396.52 total 130.13 last 63.57)
 - open/close 9327 (time 1789025406.53 total 140.15 last 67.21)
 - open/close 9942 (time 1789025416.53 total 150.15 last 61.50)
total: 10000 open/close in 150.75 seconds: 66.33 ops/second
 - unlinked 0 (time 1789025422 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 751 (time 1789025524.53 total 10.00 last 75.09)
 - open/close 1421 (time 1789025534.54 total 20.01 last 66.96)
 - open/close 2074 (time 1789025544.55 total 30.02 last 65.23)
 - open/close 2785 (time 1789025554.56 total 40.03 last 71.01)
 - open/close 3384 (time 1789025564.57 total 50.04 last 59.87)
 - open/close 4067 (time 1789025574.58 total 60.04 last 68.25)
 - open/close 4935 (time 1789025584.58 total 70.05 last 86.76)
 - open/close 5703 (time 1789025594.58 total 80.05 last 76.78)
 - open/close 6472 (time 1789025604.59 total 90.06 last 76.84)
 - open/close 7298 (time 1789025614.59 total 100.06 last 82.58)
 - open/close 7985 (time 1789025624.60 total 110.06 last 68.67)
 - open/close 8543 (time 1789025634.60 total 120.06 last 55.79)
 - open/close 9178 (time 1789025644.60 total 130.07 last 63.49)
 - open/close 9924 (time 1789025654.61 total 140.08 last 74.50)
total: 10000 open/close in 141.00 seconds: 70.92 ops/second
 - unlinked 0 (time 1789025659 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 793 (time 1789025757.56 total 10.00 last 79.26)
 - open/close 1498 (time 1789025767.57 total 20.01 last 70.43)
 - open/close 2161 (time 1789025777.57 total 30.02 last 66.26)
 - open/close 2790 (time 1789025787.58 total 40.03 last 62.83)
 - open/close 3510 (time 1789025797.59 total 50.03 last 71.98)
 - open/close 4217 (time 1789025807.59 total 60.04 last 70.65)
 - open/close 4801 (time 1789025817.62 total 70.07 last 58.24)
 - open/close 5296 (time 1789025827.63 total 80.08 last 49.46)
 - open/close 5938 (time 1789025837.63 total 90.08 last 64.17)
 - open/close 6598 (time 1789025847.64 total 100.08 last 65.98)
 - open/close 7300 (time 1789025857.64 total 110.08 last 70.19)
 - open/close 8040 (time 1789025867.64 total 120.09 last 73.97)
 - open/close 8631 (time 1789025877.65 total 130.10 last 59.05)
 - open/close 9324 (time 1789025887.67 total 140.12 last 69.14)
total: 10000 open/close in 149.53 seconds: 66.88 ops/second
 - unlinked 0 (time 1789025902 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 690 (time 1789026015.31 total 10.00 last 68.98)
 - open/close 1370 (time 1789026025.31 total 20.00 last 67.99)
 - open/close 2102 (time 1789026035.32 total 30.01 last 73.15)
 - open/close 2721 (time 1789026045.32 total 40.02 last 61.87)
 - open/close 3373 (time 1789026055.36 total 50.05 last 64.96)
 - open/close 3982 (time 1789026065.38 total 60.07 last 60.79)
 - open/close 4587 (time 1789026075.38 total 70.08 last 60.45)
 - open/close 5263 (time 1789026085.42 total 80.12 last 67.35)
 - open/close 5795 (time 1789026095.42 total 90.12 last 53.18)
 - open/close 6413 (time 1789026105.44 total 100.13 last 61.74)
 - open/close 7113 (time 1789026115.45 total 110.14 last 69.93)
 - open/close 7831 (time 1789026125.45 total 120.15 last 71.73)
 - open/close 8598 (time 1789026135.46 total 130.16 last 76.63)
 - open/close 9313 (time 1789026145.47 total 140.17 last 71.43)
total: 10000 open/close in 149.69 seconds: 66.80 ops/second
 - unlinked 0 (time 1789026159 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 679 (time 1789026262.85 total 10.00 last 67.90)
 - open/close 1275 (time 1789026272.87 total 20.02 last 59.49)
 - open/close 2024 (time 1789026282.87 total 30.02 last 74.87)
 - open/close 2679 (time 1789026292.87 total 40.02 last 65.49)
 - open/close 3371 (time 1789026302.89 total 50.04 last 69.07)
 - open/close 4009 (time 1789026312.89 total 60.04 last 63.80)
 - open/close 4690 (time 1789026322.90 total 70.05 last 68.05)
 - open/close 5414 (time 1789026332.92 total 80.08 last 72.22)
 - open/close 6089 (time 1789026342.93 total 90.08 last 67.46)
 - open/close 6681 (time 1789026352.93 total 100.08 last 59.18)
 - open/close 7149 (time 1789026362.94 total 110.09 last 46.76)
 - open/close 7773 (time 1789026372.95 total 120.10 last 62.34)
 - open/close 8486 (time 1789026382.95 total 130.10 last 71.28)
 - open/close 9226 (time 1789026392.97 total 140.12 last 73.86)
 - open/close 9826 (time 1789026402.97 total 150.12 last 60.00)
total: 10000 open/close in 152.56 seconds: 65.55 ops/second
 - unlinked 0 (time 1789026410 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg650-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg650-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg650-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.150@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.150@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg650-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1
Started lustre-OST0000
oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg650-server: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg650-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg650-client.virtnet:  -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre
On OST0, 16670 used inodes rc=0
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg650-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg650-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg650-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.150@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.150@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.150@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.150@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.150@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg650-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1
Started lustre-OST0000
oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg650-server: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg650-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg650-client.virtnet:  -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg650-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg650-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg650-server
stop mds service on oleg650-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg650-server
unloading modules via unload_modules_local on: 'oleg650-server'
oleg650-server: oleg650-server.virtnet: executing unload_modules_local
oleg650-server: kmemleak disabled
modules unloaded.
pdsh@oleg650-client: oleg650-client: ssh exited with exit code 2
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 2
pdsh@oleg650-client: oleg650-client: ssh exited with exit code 2
pdsh@oleg650-client: oleg650-server: ssh exited with exit code 2