== conf-sanity test 69: replace an OST with the same index ========================================================== 07:25:03 (1789298703)
start mds service on oleg460-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep 12 19:14 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg460-server'
oleg460-server: oleg460-server.virtnet: executing load_modules_local
oleg460-server: kmemleak disabled
oleg460-server: Loading modules from /home/green/git/lustre-release/lustre
oleg460-server: detected 4 online CPUs by sysfs
oleg460-server: MODOPTS_LIBCFS=
oleg460-server: Force libcfs to create 2 CPU partitions
oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg460-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg460-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1
Started lustre-OST0000
oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg460-client.virtnet:  -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 612 (time 1789298792.04 total 10.00 last 61.19)
 - open/close 1235 (time 1789298802.05 total 20.01 last 62.25)
 - open/close 1831 (time 1789298812.06 total 30.02 last 59.53)
 - open/close 2435 (time 1789298822.07 total 40.03 last 60.37)
 - open/close 3120 (time 1789298832.07 total 50.03 last 68.47)
 - open/close 3693 (time 1789298842.07 total 60.03 last 57.29)
 - open/close 4305 (time 1789298852.08 total 70.04 last 61.15)
 - open/close 4962 (time 1789298862.10 total 80.06 last 65.58)
 - open/close 5622 (time 1789298872.10 total 90.06 last 66.00)
 - open/close 6165 (time 1789298882.12 total 100.08 last 54.16)
 - open/close 6711 (time 1789298892.14 total 110.10 last 54.52)
 - open/close 7188 (time 1789298902.15 total 120.11 last 47.65)
 - open/close 7753 (time 1789298912.16 total 130.12 last 56.42)
 - open/close 8284 (time 1789298922.18 total 140.15 last 52.98)
 - open/close 8741 (time 1789298932.20 total 150.17 last 45.61)
 - open/close 9217 (time 1789298942.21 total 160.17 last 47.60)
 - open/close 9836 (time 1789298952.22 total 170.18 last 61.83)
total: 10000 open/close in 172.71 seconds: 57.90 ops/second
 - unlinked 0 (time 1789298959 ; total 1 ; last 1)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 624 (time 1789299080.85 total 10.01 last 62.33)
 - open/close 1224 (time 1789299090.85 total 20.02 last 59.97)
 - open/close 1851 (time 1789299100.87 total 30.04 last 62.58)
 - open/close 2446 (time 1789299110.87 total 40.04 last 59.49)
 - open/close 3077 (time 1789299120.89 total 50.05 last 63.02)
 - open/close 3688 (time 1789299130.89 total 60.05 last 61.09)
 - open/close 4340 (time 1789299140.90 total 70.06 last 65.14)
 - open/close 4949 (time 1789299150.91 total 80.07 last 60.83)
 - open/close 5583 (time 1789299160.96 total 90.13 last 63.05)
 - open/close 6201 (time 1789299170.97 total 100.14 last 61.74)
 - open/close 6913 (time 1789299180.98 total 110.15 last 71.13)
 - open/close 7579 (time 1789299190.99 total 120.16 last 66.54)
 - open/close 8176 (time 1789299201.01 total 130.17 last 59.61)
 - open/close 8740 (time 1789299211.01 total 140.18 last 56.38)
 - open/close 9247 (time 1789299221.02 total 150.19 last 50.65)
 - open/close 9846 (time 1789299231.03 total 160.19 last 59.86)
total: 10000 open/close in 162.74 seconds: 61.45 ops/second
 - unlinked 0 (time 1789299237 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 554 (time 1789299353.14 total 10.00 last 55.40)
 - open/close 1213 (time 1789299363.14 total 20.00 last 65.88)
 - open/close 1815 (time 1789299373.14 total 30.00 last 60.19)
 - open/close 2355 (time 1789299383.15 total 40.02 last 53.93)
 - open/close 2965 (time 1789299393.16 total 50.02 last 60.97)
 - open/close 3554 (time 1789299403.17 total 60.03 last 58.84)
 - open/close 4182 (time 1789299413.17 total 70.04 last 62.76)
 - open/close 4787 (time 1789299423.18 total 80.04 last 60.48)
 - open/close 5439 (time 1789299433.18 total 90.04 last 65.19)
 - open/close 6077 (time 1789299443.18 total 100.05 last 63.78)
 - open/close 6752 (time 1789299453.19 total 110.05 last 67.45)
 - open/close 7367 (time 1789299463.19 total 120.06 last 61.49)
 - open/close 7944 (time 1789299473.21 total 130.07 last 57.62)
 - open/close 8423 (time 1789299483.22 total 140.09 last 47.82)
 - open/close 9064 (time 1789299493.23 total 150.10 last 64.03)
 - open/close 9697 (time 1789299503.24 total 160.10 last 63.26)
total: 10000 open/close in 165.35 seconds: 60.48 ops/second
 - unlinked 0 (time 1789299513 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 594 (time 1789299625.34 total 10.01 last 59.35)
 - open/close 1182 (time 1789299635.36 total 20.03 last 58.67)
 - open/close 1730 (time 1789299645.37 total 30.04 last 54.75)
 - open/close 2306 (time 1789299655.38 total 40.04 last 57.58)
 - open/close 2916 (time 1789299665.38 total 50.05 last 60.95)
 - open/close 3597 (time 1789299675.41 total 60.07 last 67.94)
 - open/close 4233 (time 1789299685.42 total 70.09 last 63.51)
 - open/close 4755 (time 1789299695.43 total 80.10 last 52.15)
 - open/close 5338 (time 1789299705.44 total 90.11 last 58.24)
 - open/close 5964 (time 1789299715.45 total 100.12 last 62.50)
 - open/close 6569 (time 1789299725.46 total 110.12 last 60.48)
 - open/close 7164 (time 1789299735.47 total 120.14 last 59.43)
 - open/close 7803 (time 1789299745.47 total 130.14 last 63.88)
 - open/close 8451 (time 1789299755.48 total 140.15 last 64.76)
 - open/close 9093 (time 1789299765.49 total 150.16 last 64.13)
 - open/close 9685 (time 1789299775.51 total 160.18 last 59.05)
total: 10000 open/close in 164.17 seconds: 60.91 ops/second
 - unlinked 0 (time 1789299783 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 581 (time 1789299895.69 total 10.01 last 58.03)
 - open/close 1182 (time 1789299905.70 total 20.02 last 60.07)
 - open/close 1815 (time 1789299915.70 total 30.02 last 63.27)
 - open/close 2370 (time 1789299925.70 total 40.03 last 55.47)
 - open/close 2792 (time 1789299935.72 total 50.04 last 42.14)
 - open/close 3251 (time 1789299945.72 total 60.04 last 45.89)
 - open/close 3671 (time 1789299955.73 total 70.06 last 41.95)
 - open/close 4119 (time 1789299965.76 total 80.08 last 44.69)
 - open/close 4637 (time 1789299975.78 total 90.10 last 51.71)
 - open/close 5313 (time 1789299985.78 total 100.10 last 67.56)
 - open/close 5931 (time 1789299995.79 total 110.11 last 61.78)
 - open/close 6541 (time 1789300005.80 total 120.12 last 60.90)
 - open/close 7110 (time 1789300015.81 total 130.13 last 56.85)
 - open/close 7741 (time 1789300025.81 total 140.14 last 63.07)
 - open/close 8361 (time 1789300035.82 total 150.14 last 61.99)
 - open/close 9011 (time 1789300045.82 total 160.14 last 64.95)
 - open/close 9627 (time 1789300055.83 total 170.15 last 61.54)
total: 10000 open/close in 176.13 seconds: 56.78 ops/second
 - unlinked 0 (time 1789300066 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 702 (time 1789300180.95 total 10.01 last 70.14)
 - open/close 1349 (time 1789300190.95 total 20.01 last 64.66)
 - open/close 1931 (time 1789300200.97 total 30.03 last 58.13)
 - open/close 2427 (time 1789300210.99 total 40.05 last 49.50)
 - open/close 3063 (time 1789300220.99 total 50.05 last 63.58)
 - open/close 3685 (time 1789300231.00 total 60.06 last 62.16)
 - open/close 4246 (time 1789300241.01 total 70.07 last 56.04)
 - open/close 4893 (time 1789300251.01 total 80.07 last 64.65)
 - open/close 5488 (time 1789300261.03 total 90.09 last 59.42)
 - open/close 6130 (time 1789300271.03 total 100.09 last 64.17)
 - open/close 6764 (time 1789300281.04 total 110.10 last 63.37)
 - open/close 7415 (time 1789300291.05 total 120.11 last 65.01)
 - open/close 8080 (time 1789300301.06 total 130.12 last 66.44)
 - open/close 8711 (time 1789300311.06 total 140.12 last 63.09)
 - open/close 9321 (time 1789300321.08 total 150.14 last 60.93)
 - open/close 9958 (time 1789300331.09 total 160.15 last 63.62)
total: 10000 open/close in 160.72 seconds: 62.22 ops/second
 - unlinked 0 (time 1789300336 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 524 (time 1789300460.75 total 10.00 last 52.38)
 - open/close 1113 (time 1789300470.77 total 20.03 last 58.77)
 - open/close 1606 (time 1789300480.78 total 30.03 last 49.27)
 - open/close 2215 (time 1789300490.79 total 40.04 last 60.87)
 - open/close 2798 (time 1789300500.80 total 50.05 last 58.22)
 - open/close 3375 (time 1789300510.80 total 60.06 last 57.67)
 - open/close 3991 (time 1789300520.81 total 70.06 last 61.58)
 - open/close 4595 (time 1789300530.82 total 80.07 last 60.33)
 - open/close 5179 (time 1789300540.83 total 90.08 last 58.34)
 - open/close 5794 (time 1789300550.84 total 100.09 last 61.42)
 - open/close 6413 (time 1789300560.84 total 110.10 last 61.89)
 - open/close 7052 (time 1789300570.86 total 120.11 last 63.81)
 - open/close 7625 (time 1789300580.87 total 130.12 last 57.24)
 - open/close 8187 (time 1789300590.89 total 140.14 last 56.06)
 - open/close 8790 (time 1789300600.90 total 150.15 last 60.24)
 - open/close 9422 (time 1789300610.90 total 160.16 last 63.19)
total: 10000 open/close in 170.07 seconds: 58.80 ops/second
 - unlinked 0 (time 1789300625 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 579 (time 1789300746.86 total 10.01 last 57.83)
 - open/close 1125 (time 1789300756.86 total 20.01 last 54.60)
 - open/close 1672 (time 1789300766.87 total 30.02 last 54.66)
 - open/close 2334 (time 1789300776.87 total 40.02 last 66.19)
 - open/close 2953 (time 1789300786.88 total 50.03 last 61.86)
 - open/close 3529 (time 1789300796.89 total 60.03 last 57.57)
 - open/close 4110 (time 1789300806.91 total 70.06 last 57.94)
 - open/close 4710 (time 1789300816.91 total 80.06 last 59.99)
 - open/close 5311 (time 1789300826.92 total 90.07 last 60.06)
 - open/close 5935 (time 1789300836.92 total 100.07 last 62.38)
 - open/close 6591 (time 1789300846.94 total 110.09 last 65.51)
 - open/close 7134 (time 1789300856.95 total 120.10 last 54.25)
 - open/close 7722 (time 1789300866.96 total 130.11 last 58.72)
 - open/close 8401 (time 1789300876.96 total 140.11 last 67.89)
 - open/close 8979 (time 1789300886.97 total 150.12 last 57.76)
 - open/close 9581 (time 1789300896.99 total 160.14 last 60.09)
total: 10000 open/close in 167.43 seconds: 59.73 ops/second
 - unlinked 0 (time 1789300909 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 431 (time 1789301023.26 total 10.05 last 42.91)
 - open/close 934 (time 1789301033.26 total 20.05 last 50.28)
 - open/close 1489 (time 1789301043.29 total 30.07 last 55.36)
 - open/close 2051 (time 1789301053.30 total 40.09 last 56.11)
 - open/close 2676 (time 1789301063.31 total 50.10 last 62.43)
 - open/close 3320 (time 1789301073.33 total 60.12 last 64.27)
 - open/close 3927 (time 1789301083.34 total 70.13 last 60.65)
 - open/close 4482 (time 1789301093.35 total 80.13 last 55.46)
 - open/close 5116 (time 1789301103.37 total 90.16 last 63.26)
 - open/close 5712 (time 1789301113.39 total 100.17 last 59.50)
 - open/close 6391 (time 1789301123.40 total 110.19 last 67.77)
 - open/close 6876 (time 1789301133.40 total 120.19 last 48.50)
 - open/close 7482 (time 1789301143.42 total 130.21 last 60.51)
 - open/close 8074 (time 1789301153.44 total 140.23 last 59.05)
 - open/close 8719 (time 1789301163.45 total 150.23 last 64.48)
 - open/close 9352 (time 1789301173.46 total 160.25 last 63.23)
 - open/close 9932 (time 1789301183.47 total 170.26 last 57.92)
total: 10000 open/close in 171.09 seconds: 58.45 ops/second
 - unlinked 0 (time 1789301189 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 655 (time 1789301305.10 total 10.01 last 65.41)
 - open/close 1264 (time 1789301315.12 total 20.03 last 60.82)
 - open/close 1870 (time 1789301325.12 total 30.03 last 60.56)
 - open/close 2512 (time 1789301335.12 total 40.04 last 64.19)
 - open/close 3121 (time 1789301345.13 total 50.04 last 60.88)
 - open/close 3789 (time 1789301355.14 total 60.05 last 66.73)
 - open/close 4310 (time 1789301365.15 total 70.06 last 52.06)
 - open/close 4856 (time 1789301375.15 total 80.06 last 54.59)
 - open/close 5368 (time 1789301385.15 total 90.06 last 51.20)
 - open/close 5858 (time 1789301395.15 total 100.07 last 48.97)
 - open/close 6425 (time 1789301405.16 total 110.07 last 56.67)
 - open/close 6987 (time 1789301415.17 total 120.08 last 56.15)
 - open/close 7538 (time 1789301425.18 total 130.09 last 55.05)
 - open/close 8081 (time 1789301435.18 total 140.09 last 54.29)
 - open/close 8707 (time 1789301445.18 total 150.10 last 62.57)
 - open/close 9346 (time 1789301455.18 total 160.10 last 63.89)
 - open/close 9974 (time 1789301465.19 total 170.10 last 62.79)
total: 10000 open/close in 170.42 seconds: 58.68 ops/second
 - unlinked 0 (time 1789301469 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg460-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg460-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg460-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1
Started lustre-OST0000
oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg460-client.virtnet:  -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre
On OST0, 15854 used inodes rc=0
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg460-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg460-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.160@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.160@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg460-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1
Started lustre-OST0000
oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg460-client.virtnet:  -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg460-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg460-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server
stop mds service on oleg460-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server
unloading modules via unload_modules_local on: 'oleg460-server'
oleg460-server: oleg460-server.virtnet: executing unload_modules_local
oleg460-server: kmemleak disabled
modules unloaded.
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2
pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2
pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2
pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2