== conf-sanity test 69: replace an OST with the same index ========================================================== 09:02:37 (1790946157)
start mds service on oleg121-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1946968 Oct  2 03:41 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg121-server'
oleg121-server: oleg121-server.virtnet: executing load_modules_local
oleg121-server: kmemleak disabled
oleg121-server: Loading modules from /home/green/git/lustre-release/lustre
oleg121-server: detected 4 online CPUs by sysfs
oleg121-server: MODOPTS_LIBCFS=
oleg121-server: Force libcfs to create 2 CPU partitions
oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg121-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 630 (time 1790946251.87 total 10.00 last 62.97)
 - open/close 1328 (time 1790946261.89 total 20.02 last 69.66)
 - open/close 2059 (time 1790946271.91 total 30.04 last 73.02)
 - open/close 2798 (time 1790946281.91 total 40.04 last 73.87)
 - open/close 3536 (time 1790946291.91 total 50.04 last 73.77)
 - open/close 4360 (time 1790946301.92 total 60.05 last 82.32)
 - open/close 5037 (time 1790946311.94 total 70.07 last 67.63)
 - open/close 5691 (time 1790946321.96 total 80.09 last 65.24)
 - open/close 6372 (time 1790946331.96 total 90.09 last 68.09)
 - open/close 7052 (time 1790946341.96 total 100.09 last 67.98)
 - open/close 7729 (time 1790946351.97 total 110.10 last 67.67)
 - open/close 8483 (time 1790946361.98 total 120.11 last 75.32)
 - open/close 9003 (time 1790946372.00 total 130.13 last 51.89)
 - open/close 9562 (time 1790946382.00 total 140.13 last 55.88)
total: 10000 open/close in 148.36 seconds: 67.40 ops/second
 - unlinked 0 (time 1790946394 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 732 (time 1790946500.64 total 10.01 last 73.15)
 - open/close 1469 (time 1790946510.64 total 20.01 last 73.64)
 - open/close 2239 (time 1790946520.65 total 30.02 last 76.98)
 - open/close 2942 (time 1790946530.65 total 40.02 last 70.29)
 - open/close 3606 (time 1790946540.65 total 50.02 last 66.37)
 - open/close 4335 (time 1790946550.66 total 60.03 last 72.86)
 - open/close 4944 (time 1790946560.67 total 70.04 last 60.85)
 - open/close 5666 (time 1790946570.68 total 80.05 last 72.12)
 - open/close 6308 (time 1790946580.68 total 90.05 last 64.20)
 - open/close 6980 (time 1790946590.69 total 100.06 last 67.13)
 - open/close 7722 (time 1790946600.70 total 110.07 last 74.13)
 - open/close 8421 (time 1790946610.72 total 120.09 last 69.75)
 - open/close 9026 (time 1790946620.73 total 130.10 last 60.46)
 - open/close 9650 (time 1790946630.74 total 140.11 last 62.34)
total: 10000 open/close in 144.88 seconds: 69.02 ops/second
 - unlinked 0 (time 1790946641 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 693 (time 1790946753.48 total 10.02 last 69.17)
 - open/close 1345 (time 1790946763.49 total 20.03 last 65.12)
 - open/close 2056 (time 1790946773.50 total 30.03 last 71.07)
 - open/close 2816 (time 1790946783.50 total 40.04 last 75.97)
 - open/close 3435 (time 1790946793.50 total 50.04 last 61.88)
 - open/close 4142 (time 1790946803.51 total 60.05 last 70.64)
 - open/close 4814 (time 1790946813.52 total 70.06 last 67.13)
 - open/close 5527 (time 1790946823.53 total 80.07 last 71.24)
 - open/close 6259 (time 1790946833.53 total 90.07 last 73.17)
 - open/close 6981 (time 1790946843.54 total 100.08 last 72.15)
 - open/close 7686 (time 1790946853.55 total 110.08 last 70.47)
 - open/close 8341 (time 1790946863.56 total 120.10 last 65.39)
 - open/close 8938 (time 1790946873.58 total 130.12 last 59.61)
 - open/close 9576 (time 1790946883.58 total 140.12 last 63.79)
total: 10000 open/close in 146.42 seconds: 68.30 ops/second
 - unlinked 0 (time 1790946894 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 729 (time 1790946993.73 total 10.01 last 72.82)
 - open/close 1431 (time 1790947003.74 total 20.02 last 70.15)
 - open/close 2115 (time 1790947013.74 total 30.02 last 68.37)
 - open/close 2749 (time 1790947023.74 total 40.03 last 63.37)
 - open/close 3339 (time 1790947033.77 total 50.05 last 58.84)
 - open/close 4075 (time 1790947043.79 total 60.07 last 73.49)
 - open/close 4645 (time 1790947053.81 total 70.09 last 56.88)
 - open/close 5225 (time 1790947063.81 total 80.10 last 57.97)
 - open/close 5720 (time 1790947073.82 total 90.10 last 49.48)
 - open/close 6259 (time 1790947083.82 total 100.10 last 53.87)
 - open/close 6916 (time 1790947093.83 total 110.11 last 65.68)
 - open/close 7507 (time 1790947103.83 total 120.11 last 59.08)
 - open/close 8246 (time 1790947113.83 total 130.12 last 73.87)
 - open/close 8877 (time 1790947123.84 total 140.12 last 63.07)
 - open/close 9609 (time 1790947133.85 total 150.13 last 73.15)
total: 10000 open/close in 155.94 seconds: 64.13 ops/second
 - unlinked 0 (time 1790947144 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 601 (time 1790947251.74 total 10.01 last 60.07)
 - open/close 1199 (time 1790947261.75 total 20.01 last 59.75)
 - open/close 1817 (time 1790947271.76 total 30.03 last 61.72)
 - open/close 2307 (time 1790947281.78 total 40.04 last 48.92)
 - open/close 2927 (time 1790947291.79 total 50.05 last 61.95)
 - open/close 3644 (time 1790947301.79 total 60.06 last 71.65)
 - open/close 4369 (time 1790947311.79 total 70.06 last 72.49)
 - open/close 5104 (time 1790947321.82 total 80.09 last 73.30)
 - open/close 5853 (time 1790947331.83 total 90.09 last 74.87)
 - open/close 6523 (time 1790947341.84 total 100.11 last 66.89)
 - open/close 7201 (time 1790947351.84 total 110.11 last 67.80)
 - open/close 7952 (time 1790947361.85 total 120.11 last 75.08)
 - open/close 8687 (time 1790947371.86 total 130.13 last 73.38)
 - open/close 9345 (time 1790947381.87 total 140.14 last 65.72)
total: 10000 open/close in 149.98 seconds: 66.68 ops/second
 - unlinked 0 (time 1790947396 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 735 (time 1790947503.07 total 10.01 last 73.43)
 - open/close 1407 (time 1790947513.07 total 20.01 last 67.16)
 - open/close 2048 (time 1790947523.08 total 30.02 last 64.08)
 - open/close 2760 (time 1790947533.08 total 40.02 last 71.19)
 - open/close 3348 (time 1790947543.09 total 50.03 last 58.73)
 - open/close 3950 (time 1790947553.10 total 60.04 last 60.16)
 - open/close 4450 (time 1790947563.14 total 70.08 last 49.81)
 - open/close 5040 (time 1790947573.14 total 80.08 last 58.97)
 - open/close 5864 (time 1790947583.14 total 90.08 last 82.39)
 - open/close 6626 (time 1790947593.14 total 100.08 last 76.18)
 - open/close 7331 (time 1790947603.16 total 110.10 last 70.41)
 - open/close 8018 (time 1790947613.17 total 120.11 last 68.63)
 - open/close 8644 (time 1790947623.20 total 130.14 last 62.38)
 - open/close 9327 (time 1790947633.21 total 140.15 last 68.25)
 - open/close 9916 (time 1790947643.22 total 150.16 last 58.84)
total: 10000 open/close in 151.71 seconds: 65.92 ops/second
 - unlinked 0 (time 1790947650 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 683 (time 1790947755.44 total 10.01 last 68.26)
 - open/close 1465 (time 1790947765.46 total 20.02 last 78.10)
 - open/close 2191 (time 1790947775.46 total 30.02 last 72.57)
 - open/close 2794 (time 1790947785.47 total 40.03 last 60.27)
 - open/close 3502 (time 1790947795.47 total 50.03 last 70.75)
 - open/close 4116 (time 1790947805.48 total 60.04 last 61.36)
 - open/close 4688 (time 1790947815.50 total 70.06 last 57.11)
 - open/close 5325 (time 1790947825.50 total 80.06 last 63.68)
 - open/close 5893 (time 1790947835.50 total 90.06 last 56.79)
 - open/close 6592 (time 1790947845.50 total 100.06 last 69.88)
 - open/close 7384 (time 1790947855.50 total 110.07 last 79.19)
 - open/close 8092 (time 1790947865.51 total 120.07 last 70.78)
 - open/close 8722 (time 1790947875.52 total 130.08 last 62.92)
 - open/close 9453 (time 1790947885.53 total 140.09 last 73.05)
total: 10000 open/close in 146.72 seconds: 68.15 ops/second
 - unlinked 0 (time 1790947896 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 654 (time 1790948001.41 total 10.00 last 65.39)
 - open/close 1268 (time 1790948011.42 total 20.02 last 61.29)
 - open/close 1985 (time 1790948021.44 total 30.03 last 71.62)
 - open/close 2781 (time 1790948031.45 total 40.04 last 79.51)
 - open/close 3493 (time 1790948041.45 total 50.04 last 71.19)
 - open/close 4224 (time 1790948051.46 total 60.06 last 72.98)
 - open/close 4857 (time 1790948061.47 total 70.07 last 63.25)
 - open/close 5612 (time 1790948071.48 total 80.07 last 75.46)
 - open/close 6270 (time 1790948081.48 total 90.07 last 65.80)
 - open/close 7088 (time 1790948091.49 total 100.08 last 81.73)
 - open/close 7785 (time 1790948101.49 total 110.09 last 69.65)
 - open/close 8508 (time 1790948111.50 total 120.10 last 72.22)
 - open/close 9218 (time 1790948121.52 total 130.11 last 70.91)
 - open/close 9916 (time 1790948131.55 total 140.15 last 69.55)
total: 10000 open/close in 141.56 seconds: 70.64 ops/second
 - unlinked 0 (time 1790948138 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 692 (time 1790948244.71 total 10.01 last 69.12)
 - open/close 1473 (time 1790948254.72 total 20.02 last 78.05)
 - open/close 2223 (time 1790948264.73 total 30.03 last 74.90)
 - open/close 2983 (time 1790948274.73 total 40.04 last 75.97)
 - open/close 3773 (time 1790948284.74 total 50.04 last 78.95)
 - open/close 4507 (time 1790948294.74 total 60.05 last 73.38)
 - open/close 5149 (time 1790948304.75 total 70.05 last 64.19)
 - open/close 5862 (time 1790948314.75 total 80.05 last 71.26)
 - open/close 6557 (time 1790948324.76 total 90.06 last 69.44)
 - open/close 7212 (time 1790948334.77 total 100.07 last 65.42)
 - open/close 7955 (time 1790948344.78 total 110.08 last 74.22)
 - open/close 8682 (time 1790948354.79 total 120.09 last 72.65)
 - open/close 9420 (time 1790948364.79 total 130.09 last 73.78)
total: 10000 open/close in 137.92 seconds: 72.50 ops/second
 - unlinked 0 (time 1790948377 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 674 (time 1790948479.14 total 10.00 last 67.37)
 - open/close 1468 (time 1790948489.15 total 20.01 last 79.32)
 - open/close 2218 (time 1790948499.16 total 30.02 last 74.95)
 - open/close 2987 (time 1790948509.17 total 40.03 last 76.84)
 - open/close 3752 (time 1790948519.17 total 50.03 last 76.48)
 - open/close 4529 (time 1790948529.18 total 60.04 last 77.66)
 - open/close 5338 (time 1790948539.19 total 70.05 last 80.79)
 - open/close 5976 (time 1790948549.20 total 80.06 last 63.75)
 - open/close 6699 (time 1790948559.21 total 90.07 last 72.24)
 - open/close 7337 (time 1790948569.21 total 100.07 last 63.79)
 - open/close 8054 (time 1790948579.21 total 110.07 last 71.66)
 - open/close 8796 (time 1790948589.22 total 120.08 last 74.14)
 - open/close 9531 (time 1790948599.23 total 130.09 last 73.45)
total: 10000 open/close in 136.85 seconds: 73.08 ops/second
 - unlinked 0 (time 1790948610 ; total 1 ; last 1)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
On OST0, 14366 used inodes rc=0
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.121@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.121@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.121@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server
stop mds service on oleg121-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server
unloading modules via unload_modules_local on: 'oleg121-server'
oleg121-server: oleg121-server.virtnet: executing unload_modules_local
oleg121-server: kmemleak disabled
modules unloaded.
pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2