== conf-sanity test 69: replace an OST with the same index ========================================================== 13:06:56 (1788800816)
start mds service on oleg222-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921056 Sep  7 12:38 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg222-server'
oleg222-server: oleg222-server.virtnet: executing load_modules_local
oleg222-server: kmemleak disabled
oleg222-server: Loading modules from /home/green/git/lustre-release/lustre
oleg222-server: detected 4 online CPUs by sysfs
oleg222-server: MODOPTS_LIBCFS=
oleg222-server: Force libcfs to create 2 CPU partitions
oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg222-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg222-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1
Started lustre-OST0000
oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg222-client.virtnet:  -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 709 (time 1788800902.99 total 10.02 last 70.79)
 - open/close 1341 (time 1788800913.00 total 20.02 last 63.14)
 - open/close 1950 (time 1788800923.02 total 30.05 last 60.77)
 - open/close 2484 (time 1788800933.02 total 40.05 last 53.40)
 - open/close 3058 (time 1788800943.03 total 50.06 last 57.33)
 - open/close 3655 (time 1788800953.04 total 60.07 last 59.63)
 - open/close 4289 (time 1788800963.05 total 70.08 last 63.36)
 - open/close 5105 (time 1788800973.06 total 80.08 last 81.55)
 - open/close 5838 (time 1788800983.06 total 90.08 last 73.29)
 - open/close 6584 (time 1788800993.07 total 100.09 last 74.56)
 - open/close 7321 (time 1788801003.10 total 110.12 last 73.47)
 - open/close 7879 (time 1788801013.10 total 120.13 last 55.78)
 - open/close 8532 (time 1788801023.10 total 130.13 last 65.30)
 - open/close 9252 (time 1788801033.10 total 140.13 last 71.99)
 - open/close 9957 (time 1788801043.11 total 150.13 last 70.47)
total: 10000 open/close in 150.65 seconds: 66.38 ops/second
 - unlinked 0 (time 1788801048 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 602 (time 1788801162.17 total 10.01 last 60.14)
 - open/close 1124 (time 1788801172.18 total 20.02 last 52.13)
 - open/close 1705 (time 1788801182.18 total 30.02 last 58.10)
 - open/close 2334 (time 1788801192.19 total 40.04 last 62.81)
 - open/close 2894 (time 1788801202.20 total 50.05 last 55.94)
 - open/close 3473 (time 1788801212.22 total 60.07 last 57.79)
 - open/close 4031 (time 1788801222.24 total 70.08 last 55.73)
 - open/close 4665 (time 1788801232.25 total 80.09 last 63.30)
 - open/close 5223 (time 1788801242.27 total 90.11 last 55.71)
 - open/close 5877 (time 1788801252.30 total 100.14 last 65.19)
 - open/close 6494 (time 1788801262.30 total 110.14 last 61.69)
 - open/close 7051 (time 1788801272.31 total 120.15 last 55.64)
 - open/close 7820 (time 1788801282.31 total 130.16 last 76.89)
 - open/close 8602 (time 1788801292.32 total 140.17 last 78.11)
 - open/close 9110 (time 1788801302.33 total 150.17 last 50.76)
 - open/close 9740 (time 1788801312.34 total 160.18 last 62.96)
total: 10000 open/close in 164.21 seconds: 60.90 ops/second
 - unlinked 0 (time 1788801322 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 608 (time 1788801434.44 total 10.01 last 60.74)
 - open/close 1157 (time 1788801444.44 total 20.01 last 54.87)
 - open/close 1862 (time 1788801454.44 total 30.02 last 70.48)
 - open/close 2508 (time 1788801464.47 total 40.04 last 64.43)
 - open/close 3085 (time 1788801474.49 total 50.06 last 57.59)
 - open/close 3659 (time 1788801484.50 total 60.07 last 57.37)
 - open/close 4194 (time 1788801494.50 total 70.08 last 53.46)
 - open/close 4867 (time 1788801504.51 total 80.08 last 67.26)
 - open/close 5651 (time 1788801514.51 total 90.09 last 78.35)
 - open/close 6381 (time 1788801524.52 total 100.10 last 72.94)
 - open/close 6983 (time 1788801534.53 total 110.10 last 60.18)
 - open/close 7611 (time 1788801544.55 total 120.12 last 62.67)
 - open/close 8277 (time 1788801554.55 total 130.13 last 66.55)
 - open/close 8944 (time 1788801564.57 total 140.14 last 66.61)
 - open/close 9584 (time 1788801574.58 total 150.15 last 63.94)
total: 10000 open/close in 155.74 seconds: 64.21 ops/second
 - unlinked 0 (time 1788801584 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 603 (time 1788801703.98 total 10.03 last 60.14)
 - open/close 1190 (time 1788801713.99 total 20.04 last 58.64)
 - open/close 1882 (time 1788801724.00 total 30.05 last 69.12)
 - open/close 2578 (time 1788801734.01 total 40.06 last 69.51)
 - open/close 3190 (time 1788801744.02 total 50.06 last 61.19)
 - open/close 3820 (time 1788801754.03 total 60.08 last 62.88)
 - open/close 4388 (time 1788801764.06 total 70.11 last 56.67)
 - open/close 5110 (time 1788801774.06 total 80.11 last 72.19)
 - open/close 5690 (time 1788801784.07 total 90.11 last 57.96)
 - open/close 6208 (time 1788801794.09 total 100.14 last 51.68)
 - open/close 6830 (time 1788801804.10 total 110.15 last 62.14)
 - open/close 7502 (time 1788801814.10 total 120.15 last 67.20)
 - open/close 8190 (time 1788801824.12 total 130.16 last 68.70)
 - open/close 8826 (time 1788801834.12 total 140.17 last 63.56)
 - open/close 9375 (time 1788801844.15 total 150.20 last 54.73)
total: 10000 open/close in 159.77 seconds: 62.59 ops/second
 - unlinked 0 (time 1788801858 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 603 (time 1788801971.37 total 10.02 last 60.19)
 - open/close 1230 (time 1788801981.39 total 20.03 last 62.63)
 - open/close 1767 (time 1788801991.39 total 30.03 last 53.70)
 - open/close 2295 (time 1788802001.40 total 40.04 last 52.74)
 - open/close 2868 (time 1788802011.43 total 50.07 last 57.13)
 - open/close 3615 (time 1788802021.43 total 60.08 last 74.67)
 - open/close 4240 (time 1788802031.44 total 70.08 last 62.47)
 - open/close 5021 (time 1788802041.44 total 80.08 last 78.09)
 - open/close 5584 (time 1788802051.45 total 90.09 last 56.23)
 - open/close 6174 (time 1788802061.46 total 100.10 last 58.94)
 - open/close 6829 (time 1788802071.46 total 110.11 last 65.48)
 - open/close 7469 (time 1788802081.46 total 120.11 last 63.99)
 - open/close 8016 (time 1788802091.48 total 130.12 last 54.62)
 - open/close 8632 (time 1788802101.51 total 140.15 last 61.41)
 - open/close 9270 (time 1788802111.52 total 150.17 last 63.71)
 - open/close 9950 (time 1788802121.53 total 160.17 last 67.96)
total: 10000 open/close in 161.11 seconds: 62.07 ops/second
 - unlinked 0 (time 1788802126 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 780 (time 1788802233.40 total 10.00 last 77.98)
 - open/close 1605 (time 1788802243.41 total 20.01 last 82.45)
 - open/close 2336 (time 1788802253.41 total 30.01 last 73.08)
 - open/close 2913 (time 1788802263.42 total 40.02 last 57.64)
 - open/close 3598 (time 1788802273.43 total 50.04 last 68.40)
 - open/close 4287 (time 1788802283.44 total 60.04 last 68.88)
 - open/close 5039 (time 1788802293.44 total 70.04 last 75.17)
 - open/close 5761 (time 1788802303.45 total 80.05 last 72.16)
 - open/close 6496 (time 1788802313.45 total 90.05 last 73.47)
 - open/close 7246 (time 1788802323.45 total 100.05 last 74.99)
 - open/close 8086 (time 1788802333.46 total 110.06 last 83.94)
 - open/close 8760 (time 1788802343.47 total 120.07 last 67.35)
 - open/close 9501 (time 1788802353.48 total 130.09 last 73.96)
total: 10000 open/close in 136.76 seconds: 73.12 ops/second
 - unlinked 0 (time 1788802365 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 647 (time 1788802477.85 total 10.00 last 64.70)
 - open/close 1139 (time 1788802487.87 total 20.02 last 49.08)
 - open/close 1698 (time 1788802497.88 total 30.04 last 55.84)
 - open/close 2259 (time 1788802507.90 total 40.05 last 56.00)
 - open/close 2808 (time 1788802517.90 total 50.06 last 54.88)
 - open/close 3593 (time 1788802527.91 total 60.06 last 78.45)
 - open/close 4243 (time 1788802537.92 total 70.07 last 64.95)
 - open/close 4964 (time 1788802547.93 total 80.08 last 72.03)
 - open/close 5625 (time 1788802557.93 total 90.09 last 66.07)
 - open/close 6394 (time 1788802567.93 total 100.09 last 76.89)
 - open/close 7139 (time 1788802577.94 total 110.09 last 74.44)
 - open/close 7724 (time 1788802587.95 total 120.10 last 58.45)
 - open/close 8306 (time 1788802597.98 total 130.13 last 58.03)
 - open/close 8925 (time 1788802607.98 total 140.14 last 61.87)
 - open/close 9591 (time 1788802618.00 total 150.15 last 66.48)
total: 10000 open/close in 158.27 seconds: 63.18 ops/second
 - unlinked 0 (time 1788802633 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 608 (time 1788802739.51 total 10.01 last 60.72)
 - open/close 1192 (time 1788802749.52 total 20.02 last 58.36)
 - open/close 1819 (time 1788802759.52 total 30.02 last 62.69)
 - open/close 2409 (time 1788802769.53 total 40.03 last 58.94)
 - open/close 3066 (time 1788802779.54 total 50.04 last 65.65)
 - open/close 3629 (time 1788802789.54 total 60.04 last 56.30)
 - open/close 4295 (time 1788802799.55 total 70.05 last 66.54)
 - open/close 5036 (time 1788802809.56 total 80.06 last 74.03)
 - open/close 5646 (time 1788802819.56 total 90.06 last 60.99)
 - open/close 6164 (time 1788802829.57 total 100.07 last 51.74)
 - open/close 6660 (time 1788802839.58 total 110.07 last 49.59)
 - open/close 7348 (time 1788802849.58 total 120.08 last 68.78)
 - open/close 8100 (time 1788802859.58 total 130.08 last 75.17)
 - open/close 8852 (time 1788802869.59 total 140.09 last 75.17)
 - open/close 9536 (time 1788802879.60 total 150.10 last 68.33)
total: 10000 open/close in 156.76 seconds: 63.79 ops/second
 - unlinked 0 (time 1788802891 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 688 (time 1788802996.98 total 10.01 last 68.75)
 - open/close 1443 (time 1788803006.98 total 20.01 last 75.50)
 - open/close 2165 (time 1788803016.99 total 30.02 last 72.15)
 - open/close 2885 (time 1788803027.00 total 40.02 last 71.95)
 - open/close 3502 (time 1788803037.01 total 50.03 last 61.64)
 - open/close 4102 (time 1788803047.01 total 60.04 last 59.95)
 - open/close 4683 (time 1788803057.03 total 70.06 last 57.99)
 - open/close 5288 (time 1788803067.08 total 80.10 last 60.25)
 - open/close 5701 (time 1788803077.09 total 90.11 last 41.25)
 - open/close 6244 (time 1788803087.10 total 100.12 last 54.24)
 - open/close 6856 (time 1788803097.10 total 110.13 last 61.16)
 - open/close 7408 (time 1788803107.14 total 120.16 last 55.03)
 - open/close 7982 (time 1788803117.15 total 130.17 last 57.33)
 - open/close 8657 (time 1788803127.16 total 140.18 last 67.43)
 - open/close 9335 (time 1788803137.16 total 150.19 last 67.77)
total: 10000 open/close in 159.89 seconds: 62.54 ops/second
 - unlinked 0 (time 1788803151 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 602 (time 1788803263.04 total 10.01 last 60.12)
 - open/close 1365 (time 1788803273.04 total 20.01 last 76.29)
 - open/close 1948 (time 1788803283.05 total 30.02 last 58.27)
 - open/close 2697 (time 1788803293.05 total 40.02 last 74.89)
 - open/close 3485 (time 1788803303.06 total 50.03 last 78.73)
 - open/close 4320 (time 1788803313.08 total 60.05 last 83.32)
 - open/close 5085 (time 1788803323.09 total 70.06 last 76.44)
 - open/close 5862 (time 1788803333.09 total 80.06 last 77.69)
 - open/close 6507 (time 1788803343.09 total 90.07 last 64.48)
 - open/close 7199 (time 1788803353.11 total 100.08 last 69.12)
 - open/close 7879 (time 1788803363.12 total 110.09 last 67.93)
 - open/close 8602 (time 1788803373.13 total 120.10 last 72.22)
 - open/close 9327 (time 1788803383.14 total 130.11 last 72.44)
total: 10000 open/close in 140.11 seconds: 71.37 ops/second
 - unlinked 0 (time 1788803397 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg222-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg222-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg222-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1
Started lustre-OST0000
oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg222-client.virtnet:  -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre
On OST0, 13070 used inodes rc=0
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg222-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg222-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.122@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg222-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1
Started lustre-OST0000
oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg222-client.virtnet:  -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg222-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg222-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server
stop mds service on oleg222-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server
unloading modules via unload_modules_local on: 'oleg222-server'
oleg222-server: oleg222-server.virtnet: executing unload_modules_local
oleg222-server: kmemleak disabled
modules unloaded.
pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2
pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2
pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2