== conf-sanity test 69: replace an OST with the same index ========================================================== 13:32:33 (1788802353)
start mds service on oleg158-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep  7 11:22 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg158-server'
oleg158-server: oleg158-server.virtnet: executing load_modules_local
oleg158-server: kmemleak disabled
oleg158-server: Loading modules from /home/green/git/lustre-release/lustre
oleg158-server: detected 4 online CPUs by sysfs
oleg158-server: MODOPTS_LIBCFS=
oleg158-server: Force libcfs to create 2 CPU partitions
oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg158-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg158-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg158-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1
Started lustre-OST0000
oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg158-client.virtnet:  -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 698 (time 1788802442.36 total 10.01 last 69.75)
 - open/close 1403 (time 1788802452.36 total 20.01 last 70.46)
 - open/close 2282 (time 1788802462.37 total 30.02 last 87.84)
 - open/close 3108 (time 1788802472.38 total 40.04 last 82.47)
 - open/close 3854 (time 1788802482.39 total 50.04 last 74.55)
 - open/close 4733 (time 1788802492.40 total 60.05 last 87.83)
 - open/close 5626 (time 1788802502.40 total 70.05 last 89.27)
 - open/close 6385 (time 1788802512.41 total 80.06 last 75.87)
 - open/close 7234 (time 1788802522.41 total 90.06 last 84.89)
 - open/close 8064 (time 1788802532.42 total 100.07 last 82.90)
 - open/close 8931 (time 1788802542.42 total 110.07 last 86.69)
 - open/close 9796 (time 1788802552.43 total 120.08 last 86.45)
total: 10000 open/close in 122.50 seconds: 81.63 ops/second
 - unlinked 0 (time 1788802558 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 897 (time 1788802650.67 total 10.00 last 89.70)
 - open/close 1785 (time 1788802660.68 total 20.01 last 88.70)
 - open/close 2584 (time 1788802670.69 total 30.01 last 79.88)
 - open/close 3319 (time 1788802680.69 total 40.02 last 73.46)
 - open/close 4142 (time 1788802690.69 total 50.02 last 82.28)
 - open/close 4974 (time 1788802700.71 total 60.04 last 83.05)
 - open/close 5619 (time 1788802710.71 total 70.04 last 64.48)
 - open/close 6335 (time 1788802720.72 total 80.04 last 71.59)
 - open/close 7138 (time 1788802730.73 total 90.05 last 80.21)
 - open/close 7971 (time 1788802740.73 total 100.06 last 83.26)
 - open/close 8772 (time 1788802750.73 total 110.06 last 80.09)
 - open/close 9552 (time 1788802760.74 total 120.07 last 77.92)
total: 10000 open/close in 125.75 seconds: 79.52 ops/second
 - unlinked 0 (time 1788802770 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 760 (time 1788802875.21 total 10.01 last 75.92)
 - open/close 1586 (time 1788802885.21 total 20.02 last 82.54)
 - open/close 2477 (time 1788802895.22 total 30.03 last 89.03)
 - open/close 3399 (time 1788802905.22 total 40.03 last 92.20)
 - open/close 4283 (time 1788802915.23 total 50.03 last 88.36)
 - open/close 5132 (time 1788802925.23 total 60.03 last 84.87)
 - open/close 5914 (time 1788802935.23 total 70.04 last 78.16)
 - open/close 6688 (time 1788802945.25 total 80.05 last 77.30)
 - open/close 7437 (time 1788802955.26 total 90.06 last 74.81)
 - open/close 8153 (time 1788802965.28 total 100.08 last 71.48)
 - open/close 8874 (time 1788802975.28 total 110.08 last 72.07)
 - open/close 9552 (time 1788802985.30 total 120.10 last 67.69)
total: 10000 open/close in 127.16 seconds: 78.64 ops/second
 - unlinked 0 (time 1788802996 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 843 (time 1788803091.42 total 10.00 last 84.26)
 - open/close 1665 (time 1788803101.43 total 20.01 last 82.18)
 - open/close 2459 (time 1788803111.43 total 30.01 last 79.37)
 - open/close 3303 (time 1788803121.44 total 40.02 last 84.31)
 - open/close 4145 (time 1788803131.44 total 50.02 last 84.19)
 - open/close 4960 (time 1788803141.45 total 60.03 last 81.43)
 - open/close 5798 (time 1788803151.45 total 70.03 last 83.76)
 - open/close 6560 (time 1788803161.47 total 80.05 last 76.11)
 - open/close 7327 (time 1788803171.47 total 90.05 last 76.69)
 - open/close 8175 (time 1788803181.48 total 100.06 last 84.73)
 - open/close 8926 (time 1788803191.48 total 110.06 last 75.05)
 - open/close 9600 (time 1788803201.49 total 120.07 last 67.35)
total: 10000 open/close in 125.03 seconds: 79.98 ops/second
 - unlinked 0 (time 1788803210 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 891 (time 1788803299.62 total 10.01 last 89.04)
 - open/close 1734 (time 1788803309.63 total 20.02 last 84.23)
 - open/close 2531 (time 1788803319.64 total 30.03 last 79.61)
 - open/close 3324 (time 1788803329.65 total 40.04 last 79.21)
 - open/close 4170 (time 1788803339.65 total 50.04 last 84.59)
 - open/close 5073 (time 1788803349.66 total 60.04 last 90.26)
 - open/close 5919 (time 1788803359.66 total 70.05 last 84.58)
 - open/close 6753 (time 1788803369.66 total 80.05 last 83.36)
 - open/close 7436 (time 1788803379.67 total 90.06 last 68.24)
 - open/close 8278 (time 1788803389.68 total 100.07 last 84.11)
 - open/close 9039 (time 1788803399.69 total 110.08 last 76.04)
 - open/close 9820 (time 1788803409.70 total 120.09 last 78.01)
total: 10000 open/close in 122.10 seconds: 81.90 ops/second
 - unlinked 0 (time 1788803415 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 840 (time 1788803510.63 total 10.00 last 83.96)
 - open/close 1691 (time 1788803520.64 total 20.01 last 85.04)
 - open/close 2580 (time 1788803530.64 total 30.01 last 88.88)
 - open/close 3435 (time 1788803540.65 total 40.02 last 85.47)
 - open/close 4288 (time 1788803550.65 total 50.02 last 85.25)
 - open/close 5136 (time 1788803560.65 total 60.02 last 84.78)
 - open/close 5904 (time 1788803570.66 total 70.03 last 76.73)
 - open/close 6725 (time 1788803580.67 total 80.04 last 82.03)
 - open/close 7580 (time 1788803590.68 total 90.05 last 85.47)
 - open/close 8414 (time 1788803600.68 total 100.05 last 83.39)
 - open/close 9323 (time 1788803610.69 total 110.06 last 90.75)
total: 10000 open/close in 118.72 seconds: 84.23 ops/second
 - unlinked 0 (time 1788803623 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 856 (time 1788803726.29 total 10.00 last 85.59)
 - open/close 1679 (time 1788803736.29 total 20.00 last 82.30)
 - open/close 2576 (time 1788803746.30 total 30.02 last 89.58)
 - open/close 3342 (time 1788803756.31 total 40.02 last 76.53)
 - open/close 4167 (time 1788803766.31 total 50.02 last 82.50)
 - open/close 5090 (time 1788803776.31 total 60.03 last 92.28)
 - open/close 5977 (time 1788803786.32 total 70.04 last 88.59)
 - open/close 6907 (time 1788803796.34 total 80.05 last 92.88)
 - open/close 7708 (time 1788803806.34 total 90.06 last 80.05)
 - open/close 8607 (time 1788803816.34 total 100.06 last 89.89)
 - open/close 9505 (time 1788803826.35 total 110.06 last 89.79)
total: 10000 open/close in 115.48 seconds: 86.59 ops/second
 - unlinked 0 (time 1788803836 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 797 (time 1788803927.84 total 10.00 last 79.67)
 - open/close 1590 (time 1788803937.85 total 20.01 last 79.22)
 - open/close 2390 (time 1788803947.86 total 30.02 last 79.97)
 - open/close 3371 (time 1788803957.87 total 40.03 last 98.02)
 - open/close 4242 (time 1788803967.87 total 50.03 last 87.03)
 - open/close 5057 (time 1788803977.88 total 60.04 last 81.42)
 - open/close 5899 (time 1788803987.89 total 70.05 last 84.15)
 - open/close 6836 (time 1788803997.89 total 80.05 last 93.65)
 - open/close 7721 (time 1788804007.92 total 90.08 last 88.32)
 - open/close 8415 (time 1788804017.92 total 100.08 last 69.38)
 - open/close 9277 (time 1788804027.92 total 110.08 last 86.18)
total: 10000 open/close in 118.68 seconds: 84.26 ops/second
 - unlinked 0 (time 1788804040 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 960 (time 1788804132.01 total 10.01 last 95.93)
 - open/close 1840 (time 1788804142.03 total 20.03 last 87.84)
 - open/close 2418 (time 1788804152.03 total 30.03 last 57.77)
 - open/close 3315 (time 1788804162.06 total 40.06 last 89.43)
 - open/close 3965 (time 1788804172.07 total 50.07 last 64.96)
 - open/close 4800 (time 1788804182.08 total 60.07 last 83.43)
 - open/close 5607 (time 1788804192.08 total 70.08 last 80.68)
 - open/close 6464 (time 1788804202.09 total 80.08 last 85.64)
 - open/close 7346 (time 1788804212.09 total 90.09 last 88.13)
 - open/close 7952 (time 1788804222.11 total 100.11 last 60.51)
 - open/close 8546 (time 1788804232.11 total 110.11 last 59.37)
 - open/close 9332 (time 1788804242.12 total 120.11 last 78.58)
 - open/close 9952 (time 1788804252.13 total 130.13 last 61.89)
total: 10000 open/close in 130.70 seconds: 76.51 ops/second
 - unlinked 0 (time 1788804258 ; total 1 ; last 1)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 855 (time 1788804356.39 total 10.01 last 85.44)
 - open/close 1783 (time 1788804366.39 total 20.01 last 92.74)
 - open/close 2736 (time 1788804376.40 total 30.02 last 95.28)
 - open/close 3634 (time 1788804386.42 total 40.03 last 89.63)
 - open/close 4297 (time 1788804396.43 total 50.05 last 66.20)
 - open/close 4929 (time 1788804406.45 total 60.07 last 63.05)
 - open/close 5640 (time 1788804416.47 total 70.09 last 70.99)
 - open/close 6322 (time 1788804426.47 total 80.09 last 68.19)
 - open/close 7018 (time 1788804436.48 total 90.10 last 69.51)
 - open/close 7841 (time 1788804446.49 total 100.11 last 82.22)
 - open/close 8642 (time 1788804456.50 total 110.12 last 80.01)
 - open/close 9596 (time 1788804466.50 total 120.12 last 95.40)
total: 10000 open/close in 125.68 seconds: 79.57 ops/second
 - unlinked 0 (time 1788804476 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg158-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg158-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.158@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg158-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1
Started lustre-OST0000
oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg158-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg158-client.virtnet:  -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg158-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg158-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.158@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.158@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.158@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.158@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg158-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1
Started lustre-OST0000
oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg158-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg158-client.virtnet:  -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg158-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg158-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server
stop mds service on oleg158-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server
unloading modules via unload_modules_local on: 'oleg158-server'
oleg158-server: oleg158-server.virtnet: executing unload_modules_local
oleg158-server: kmemleak disabled
modules unloaded.
pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2
pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2
pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2