== conf-sanity test 69: replace an OST with the same index ========================================================== 08:56:35 (1788440195)
start mds service on oleg304-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg304-server'
oleg304-server: oleg304-server.virtnet: executing load_modules_local
oleg304-server: kmemleak disabled
oleg304-server: Loading modules from /home/green/git/lustre-release/lustre
oleg304-server: detected 4 online CPUs by sysfs
oleg304-server: MODOPTS_LIBCFS=
oleg304-server: Force libcfs to create 2 CPU partitions
oleg304-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg304-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg304-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg304-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg304-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-OST0000
oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg304-client.virtnet:  -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 582 (time 1788440286.41 total 10.00 last 58.19)
 - open/close 1184 (time 1788440296.42 total 20.01 last 60.17)
 - open/close 1867 (time 1788440306.44 total 30.03 last 68.15)
 - open/close 2537 (time 1788440316.46 total 40.05 last 66.84)
 - open/close 3115 (time 1788440326.47 total 50.06 last 57.76)
 - open/close 3730 (time 1788440336.49 total 60.08 last 61.38)
 - open/close 4360 (time 1788440346.49 total 70.08 last 63.00)
 - open/close 5040 (time 1788440356.52 total 80.11 last 67.77)
 - open/close 5688 (time 1788440366.53 total 90.12 last 64.76)
 - open/close 6354 (time 1788440376.55 total 100.14 last 66.46)
 - open/close 6989 (time 1788440386.57 total 110.16 last 63.40)
 - open/close 7561 (time 1788440396.57 total 120.16 last 57.16)
 - open/close 8259 (time 1788440406.59 total 130.18 last 69.71)
 - open/close 8893 (time 1788440416.59 total 140.18 last 63.38)
 - open/close 9539 (time 1788440426.59 total 150.18 last 64.58)
total: 10000 open/close in 158.09 seconds: 63.26 ops/second
 - unlinked 0 (time 1788440439 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 664 (time 1788440557.77 total 10.02 last 66.26)
 - open/close 1317 (time 1788440567.77 total 20.02 last 65.28)
 - open/close 1965 (time 1788440577.79 total 30.04 last 64.70)
 - open/close 2592 (time 1788440587.79 total 40.05 last 62.65)
 - open/close 3238 (time 1788440597.81 total 50.06 last 64.51)
 - open/close 3864 (time 1788440607.81 total 60.07 last 62.57)
 - open/close 4477 (time 1788440617.82 total 70.08 last 61.24)
 - open/close 5123 (time 1788440627.82 total 80.08 last 64.60)
 - open/close 5791 (time 1788440637.84 total 90.09 last 66.69)
 - open/close 6374 (time 1788440647.86 total 100.11 last 58.18)
 - open/close 7052 (time 1788440657.86 total 110.12 last 67.77)
 - open/close 7607 (time 1788440667.87 total 120.13 last 55.45)
 - open/close 8240 (time 1788440677.88 total 130.13 last 63.28)
 - open/close 8837 (time 1788440687.88 total 140.13 last 59.69)
 - open/close 9437 (time 1788440697.88 total 150.13 last 59.97)
total: 10000 open/close in 158.34 seconds: 63.16 ops/second
 - unlinked 0 (time 1788440710 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 668 (time 1788440821.50 total 10.02 last 66.65)
 - open/close 1272 (time 1788440831.51 total 20.03 last 60.35)
 - open/close 1886 (time 1788440841.52 total 30.05 last 61.30)
 - open/close 2521 (time 1788440851.53 total 40.05 last 63.49)
 - open/close 3148 (time 1788440861.53 total 50.05 last 62.68)
 - open/close 3792 (time 1788440871.53 total 60.05 last 64.40)
 - open/close 4427 (time 1788440881.54 total 70.06 last 63.43)
 - open/close 5066 (time 1788440891.54 total 80.07 last 63.88)
 - open/close 5717 (time 1788440901.55 total 90.08 last 65.04)
 - open/close 6413 (time 1788440911.56 total 100.09 last 69.53)
 - open/close 7022 (time 1788440921.57 total 110.10 last 60.84)
 - open/close 7705 (time 1788440931.57 total 120.10 last 68.30)
 - open/close 8386 (time 1788440941.58 total 130.10 last 68.05)
 - open/close 9017 (time 1788440951.59 total 140.11 last 63.05)
 - open/close 9545 (time 1788440961.61 total 150.13 last 52.69)
total: 10000 open/close in 157.66 seconds: 63.43 ops/second
 - unlinked 0 (time 1788440973 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 697 (time 1788441080.42 total 10.00 last 69.68)
 - open/close 1385 (time 1788441090.44 total 20.02 last 68.69)
 - open/close 2090 (time 1788441100.45 total 30.02 last 70.47)
 - open/close 2703 (time 1788441110.45 total 40.03 last 61.25)
 - open/close 3279 (time 1788441120.46 total 50.03 last 57.58)
 - open/close 3736 (time 1788441130.47 total 60.04 last 45.65)
 - open/close 4219 (time 1788441140.48 total 70.06 last 48.23)
 - open/close 4762 (time 1788441150.49 total 80.07 last 54.25)
 - open/close 5385 (time 1788441160.50 total 90.08 last 62.23)
 - open/close 6084 (time 1788441170.52 total 100.09 last 69.80)
 - open/close 6759 (time 1788441180.52 total 110.10 last 67.48)
 - open/close 7389 (time 1788441190.54 total 120.12 last 62.86)
 - open/close 7885 (time 1788441200.55 total 130.13 last 49.56)
 - open/close 8377 (time 1788441210.56 total 140.14 last 49.16)
 - open/close 9034 (time 1788441220.56 total 150.14 last 65.67)
 - open/close 9694 (time 1788441230.58 total 160.15 last 65.93)
total: 10000 open/close in 163.83 seconds: 61.04 ops/second
 - unlinked 0 (time 1788441238 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 623 (time 1788441347.95 total 10.01 last 62.25)
 - open/close 1219 (time 1788441357.96 total 20.02 last 59.55)
 - open/close 1928 (time 1788441367.97 total 30.02 last 70.85)
 - open/close 2586 (time 1788441377.97 total 40.03 last 65.77)
 - open/close 3317 (time 1788441387.99 total 50.04 last 72.98)
 - open/close 4033 (time 1788441397.99 total 60.05 last 71.60)
 - open/close 4676 (time 1788441407.99 total 70.05 last 64.28)
 - open/close 5330 (time 1788441418.00 total 80.06 last 65.35)
 - open/close 5958 (time 1788441428.00 total 90.06 last 62.78)
 - open/close 6578 (time 1788441438.01 total 100.07 last 61.95)
 - open/close 7291 (time 1788441448.01 total 110.07 last 71.29)
 - open/close 7892 (time 1788441458.02 total 120.08 last 60.03)
 - open/close 8621 (time 1788441468.04 total 130.09 last 72.80)
 - open/close 9307 (time 1788441478.04 total 140.10 last 68.57)
total: 10000 open/close in 149.87 seconds: 66.73 ops/second
 - unlinked 0 (time 1788441492 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 665 (time 1788441598.95 total 10.00 last 66.49)
 - open/close 1322 (time 1788441608.97 total 20.03 last 65.54)
 - open/close 1931 (time 1788441618.98 total 30.03 last 60.85)
 - open/close 2541 (time 1788441628.99 total 40.05 last 60.93)
 - open/close 3190 (time 1788441639.01 total 50.06 last 64.81)
 - open/close 3655 (time 1788441649.01 total 60.06 last 46.49)
 - open/close 4187 (time 1788441659.02 total 70.07 last 53.16)
 - open/close 4834 (time 1788441669.02 total 80.07 last 64.68)
 - open/close 5411 (time 1788441679.03 total 90.08 last 57.66)
 - open/close 6091 (time 1788441689.03 total 100.08 last 67.97)
 - open/close 6777 (time 1788441699.04 total 110.09 last 68.54)
 - open/close 7481 (time 1788441709.04 total 120.09 last 70.39)
 - open/close 8144 (time 1788441719.05 total 130.11 last 66.22)
 - open/close 8725 (time 1788441729.06 total 140.12 last 58.04)
 - open/close 9337 (time 1788441739.07 total 150.13 last 61.14)
 - open/close 9957 (time 1788441749.08 total 160.13 last 61.96)
total: 10000 open/close in 160.75 seconds: 62.21 ops/second
 - unlinked 0 (time 1788441754 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 493 (time 1788441868.86 total 10.01 last 49.26)
 - open/close 1093 (time 1788441878.87 total 20.01 last 59.97)
 - open/close 1695 (time 1788441888.88 total 30.02 last 60.14)
 - open/close 2360 (time 1788441898.88 total 40.03 last 66.48)
 - open/close 2973 (time 1788441908.92 total 50.07 last 61.05)
 - open/close 3587 (time 1788441918.93 total 60.08 last 61.34)
 - open/close 4237 (time 1788441928.96 total 70.10 last 64.83)
 - open/close 4941 (time 1788441938.97 total 80.12 last 70.27)
 - open/close 5579 (time 1788441948.99 total 90.13 last 63.73)
 - open/close 6221 (time 1788441958.99 total 100.14 last 64.14)
 - open/close 6884 (time 1788441969.00 total 110.14 last 66.28)
 - open/close 7472 (time 1788441979.03 total 120.17 last 58.63)
 - open/close 8116 (time 1788441989.03 total 130.18 last 64.34)
 - open/close 8746 (time 1788441999.04 total 140.19 last 62.94)
 - open/close 9357 (time 1788442009.06 total 150.20 last 61.04)
total: 10000 open/close in 159.81 seconds: 62.58 ops/second
 - unlinked 0 (time 1788442023 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 650 (time 1788442130.11 total 10.00 last 64.98)
 - open/close 1287 (time 1788442140.12 total 20.01 last 63.65)
 - open/close 1965 (time 1788442150.13 total 30.02 last 67.75)
 - open/close 2664 (time 1788442160.13 total 40.02 last 69.89)
 - open/close 3295 (time 1788442170.13 total 50.02 last 63.08)
 - open/close 3978 (time 1788442180.15 total 60.04 last 68.22)
 - open/close 4669 (time 1788442190.15 total 70.04 last 69.06)
 - open/close 5279 (time 1788442200.15 total 80.04 last 60.98)
 - open/close 5980 (time 1788442210.16 total 90.05 last 70.04)
 - open/close 6702 (time 1788442220.17 total 100.06 last 72.15)
 - open/close 7374 (time 1788442230.19 total 110.08 last 67.03)
 - open/close 8049 (time 1788442240.20 total 120.09 last 67.46)
 - open/close 8761 (time 1788442250.22 total 130.11 last 71.04)
 - open/close 9470 (time 1788442260.25 total 140.14 last 70.70)
total: 10000 open/close in 148.03 seconds: 67.55 ops/second
 - unlinked 0 (time 1788442272 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 689 (time 1788442383.88 total 10.01 last 68.84)
 - open/close 1365 (time 1788442393.89 total 20.02 last 67.52)
 - open/close 2065 (time 1788442403.90 total 30.03 last 69.95)
 - open/close 2740 (time 1788442413.91 total 40.04 last 67.44)
 - open/close 3358 (time 1788442423.92 total 50.04 last 61.77)
 - open/close 3997 (time 1788442433.93 total 60.05 last 63.85)
 - open/close 4680 (time 1788442443.94 total 70.06 last 68.23)
 - open/close 5271 (time 1788442453.94 total 80.06 last 59.10)
 - open/close 5942 (time 1788442463.94 total 90.07 last 67.05)
 - open/close 6531 (time 1788442473.95 total 100.07 last 58.88)
 - open/close 7131 (time 1788442483.95 total 110.08 last 59.95)
 - open/close 7837 (time 1788442493.96 total 120.09 last 70.54)
 - open/close 8521 (time 1788442503.97 total 130.10 last 68.35)
 - open/close 9230 (time 1788442513.98 total 140.10 last 70.84)
 - open/close 9949 (time 1788442523.98 total 150.11 last 71.88)
total: 10000 open/close in 150.81 seconds: 66.31 ops/second
 - unlinked 0 (time 1788442529 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 618 (time 1788442642.36 total 10.01 last 61.75)
 - open/close 1258 (time 1788442652.38 total 20.03 last 63.84)
 - open/close 1941 (time 1788442662.40 total 30.05 last 68.17)
 - open/close 2580 (time 1788442672.41 total 40.06 last 63.85)
 - open/close 3152 (time 1788442682.43 total 50.08 last 57.10)
 - open/close 3819 (time 1788442692.44 total 60.09 last 66.62)
 - open/close 4439 (time 1788442702.47 total 70.12 last 61.83)
 - open/close 5106 (time 1788442712.47 total 80.12 last 66.68)
 - open/close 5762 (time 1788442722.48 total 90.13 last 65.53)
 - open/close 6380 (time 1788442732.50 total 100.15 last 61.70)
 - open/close 7083 (time 1788442742.50 total 110.15 last 70.26)
 - open/close 7760 (time 1788442752.51 total 120.16 last 67.65)
 - open/close 8454 (time 1788442762.53 total 130.18 last 69.26)
 - open/close 9074 (time 1788442772.55 total 140.20 last 61.90)
 - open/close 9718 (time 1788442782.56 total 150.21 last 64.34)
total: 10000 open/close in 154.08 seconds: 64.90 ops/second
 - unlinked 0 (time 1788442790 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg304-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg304-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.104@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg304-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-OST0000
oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg304-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg304-client.virtnet:  -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre
On OST0, 16702 used inodes rc=0
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg304-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg304-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.104@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.104@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.104@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.104@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg304-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-OST0000
oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg304-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg304-client.virtnet:  -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg304-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg304-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server
stop mds service on oleg304-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server
unloading modules via unload_modules_local on: 'oleg304-server'
oleg304-server: oleg304-server.virtnet: executing unload_modules_local
oleg304-server: kmemleak disabled
modules unloaded.
pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2
pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2