== conf-sanity test 69: replace an OST with the same index ========================================================== 05:00:21 (1789635621)
start mds service on oleg301-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep 17 04:27 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg301-server'
oleg301-server: oleg301-server.virtnet: executing load_modules_local
oleg301-server: kmemleak disabled
oleg301-server: Loading modules from /home/green/git/lustre-release/lustre
oleg301-server: detected 4 online CPUs by sysfs
oleg301-server: MODOPTS_LIBCFS=
oleg301-server: Force libcfs to create 2 CPU partitions
oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg301-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg301-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-OST0000
oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg301-client.virtnet:  -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 801 (time 1789635703.62 total 10.00 last 80.08)
 - open/close 1646 (time 1789635713.64 total 20.02 last 84.35)
 - open/close 2380 (time 1789635723.65 total 30.03 last 73.33)
 - open/close 3179 (time 1789635733.66 total 40.04 last 79.83)
 - open/close 4027 (time 1789635743.66 total 50.04 last 84.79)
 - open/close 4864 (time 1789635753.68 total 60.07 last 83.47)
 - open/close 5683 (time 1789635763.69 total 70.07 last 81.87)
 - open/close 6437 (time 1789635773.69 total 80.07 last 75.37)
 - open/close 7181 (time 1789635783.69 total 90.08 last 74.39)
 - open/close 8006 (time 1789635793.69 total 100.08 last 82.49)
 - open/close 8763 (time 1789635803.70 total 110.08 last 75.68)
 - open/close 9547 (time 1789635813.70 total 120.08 last 78.37)
total: 10000 open/close in 125.47 seconds: 79.70 ops/second
 - unlinked 0 (time 1789635822 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 712 (time 1789635911.83 total 10.00 last 71.20)
 - open/close 1523 (time 1789635921.83 total 20.00 last 81.07)
 - open/close 2309 (time 1789635931.84 total 30.01 last 78.55)
 - open/close 3125 (time 1789635941.84 total 40.01 last 81.58)
 - open/close 3956 (time 1789635951.86 total 50.03 last 82.93)
 - open/close 4724 (time 1789635961.88 total 60.05 last 76.67)
 - open/close 5477 (time 1789635971.88 total 70.05 last 75.29)
 - open/close 6324 (time 1789635981.89 total 80.06 last 84.64)
 - open/close 7192 (time 1789635991.90 total 90.07 last 86.67)
 - open/close 8036 (time 1789636001.91 total 100.07 last 84.39)
 - open/close 8890 (time 1789636011.91 total 110.08 last 85.35)
 - open/close 9763 (time 1789636021.91 total 120.08 last 87.27)
total: 10000 open/close in 122.70 seconds: 81.50 ops/second
 - unlinked 0 (time 1789636027 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 817 (time 1789636113.45 total 10.01 last 81.64)
 - open/close 1666 (time 1789636123.45 total 20.01 last 84.90)
 - open/close 2531 (time 1789636133.46 total 30.01 last 86.45)
 - open/close 3423 (time 1789636143.46 total 40.02 last 89.16)
 - open/close 4299 (time 1789636153.46 total 50.02 last 87.56)
 - open/close 5237 (time 1789636163.47 total 60.03 last 93.78)
 - open/close 6121 (time 1789636173.47 total 70.03 last 88.35)
 - open/close 7008 (time 1789636183.47 total 80.03 last 88.69)
 - open/close 7944 (time 1789636193.47 total 90.03 last 93.60)
 - open/close 8782 (time 1789636203.49 total 100.04 last 83.69)
 - open/close 9701 (time 1789636213.49 total 110.05 last 91.86)
total: 10000 open/close in 113.16 seconds: 88.37 ops/second
 - unlinked 0 (time 1789636219 ; total 0 ; last 0)
total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second
 - open/close 850 (time 1789636301.26 total 10.01 last 84.95)
 - open/close 1719 (time 1789636311.27 total 20.01 last 86.85)
 - open/close 2558 (time 1789636321.27 total 30.02 last 83.84)
 - open/close 3214 (time 1789636331.27 total 40.02 last 65.59)
 - open/close 4054 (time 1789636341.28 total 50.02 last 83.97)
 - open/close 4938 (time 1789636351.28 total 60.03 last 88.38)
 - open/close 5894 (time 1789636361.29 total 70.03 last 95.53)
 - open/close 6797 (time 1789636371.29 total 80.03 last 90.27)
 - open/close 7748 (time 1789636381.29 total 90.04 last 95.07)
 - open/close 8731 (time 1789636391.31 total 100.05 last 98.18)
 - open/close 9679 (time 1789636401.31 total 110.05 last 94.77)
total: 10000 open/close in 113.58 seconds: 88.05 ops/second
 - unlinked 0 (time 1789636408 ; total 1 ; last 1)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 1040 (time 1789636482.64 total 10.00 last 103.96)
 - open/close 2053 (time 1789636492.65 total 20.01 last 101.21)
 - open/close 2983 (time 1789636502.66 total 30.02 last 92.89)
 - open/close 3955 (time 1789636512.67 total 40.03 last 97.11)
 - open/close 4986 (time 1789636522.67 total 50.04 last 103.04)
 - open/close 5982 (time 1789636532.68 total 60.04 last 99.55)
 - open/close 6847 (time 1789636542.68 total 70.05 last 86.47)
 - open/close 7754 (time 1789636552.69 total 80.05 last 90.65)
 - open/close 8745 (time 1789636562.69 total 90.06 last 99.03)
 - open/close 9759 (time 1789636572.70 total 100.07 last 101.30)
total: 10000 open/close in 102.32 seconds: 97.74 ops/second
 - unlinked 0 (time 1789636577 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 948 (time 1789636652.34 total 10.01 last 94.72)
 - open/close 1886 (time 1789636662.34 total 20.01 last 93.79)
 - open/close 2830 (time 1789636672.35 total 30.01 last 94.35)
 - open/close 3818 (time 1789636682.35 total 40.02 last 98.76)
 - open/close 4790 (time 1789636692.35 total 50.02 last 97.20)
 - open/close 5772 (time 1789636702.35 total 60.02 last 98.19)
 - open/close 6760 (time 1789636712.36 total 70.03 last 98.72)
 - open/close 7709 (time 1789636722.36 total 80.03 last 94.87)
 - open/close 8724 (time 1789636732.37 total 90.04 last 101.43)
 - open/close 9749 (time 1789636742.37 total 100.04 last 102.47)
total: 10000 open/close in 102.20 seconds: 97.84 ops/second
 - unlinked 0 (time 1789636746 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 996 (time 1789636821.03 total 10.00 last 99.57)
 - open/close 2027 (time 1789636831.03 total 20.01 last 103.03)
 - open/close 3032 (time 1789636841.04 total 30.02 last 100.43)
 - open/close 4180 (time 1789636851.04 total 40.02 last 114.79)
 - open/close 5065 (time 1789636861.05 total 50.02 last 88.44)
 - open/close 6003 (time 1789636871.06 total 60.04 last 93.66)
 - open/close 6921 (time 1789636881.07 total 70.05 last 91.74)
 - open/close 7862 (time 1789636891.07 total 80.05 last 94.07)
 - open/close 8700 (time 1789636901.08 total 90.05 last 83.76)
 - open/close 9626 (time 1789636911.08 total 100.06 last 92.54)
total: 10000 open/close in 103.36 seconds: 96.75 ops/second
 - unlinked 0 (time 1789636916 ; total 0 ; last 0)
total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second
 - open/close 1330 (time 1789636989.37 total 10.00 last 132.96)
 - open/close 2541 (time 1789636999.37 total 20.01 last 121.07)
 - open/close 3574 (time 1789637009.37 total 30.01 last 103.29)
 - open/close 4636 (time 1789637019.37 total 40.01 last 106.18)
 - open/close 5740 (time 1789637029.38 total 50.01 last 110.34)
 - open/close 6704 (time 1789637039.39 total 60.03 last 96.27)
 - open/close 7698 (time 1789637049.40 total 70.04 last 99.32)
 - open/close 8819 (time 1789637059.40 total 80.04 last 112.08)
 - open/close 9905 (time 1789637069.41 total 90.04 last 108.54)
total: 10000 open/close in 90.88 seconds: 110.04 ops/second
 - unlinked 0 (time 1789637072 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 1082 (time 1789637144.54 total 10.00 last 108.18)
 - open/close 2218 (time 1789637154.55 total 20.01 last 113.50)
 - open/close 3540 (time 1789637164.56 total 30.02 last 132.08)
 - open/close 4624 (time 1789637174.57 total 40.03 last 108.27)
 - open/close 5705 (time 1789637184.57 total 50.03 last 108.10)
 - open/close 6828 (time 1789637194.58 total 60.03 last 112.27)
 - open/close 8022 (time 1789637204.58 total 70.04 last 119.31)
 - open/close 9215 (time 1789637214.59 total 80.05 last 119.20)
total: 10000 open/close in 88.00 seconds: 113.63 ops/second
 - unlinked 0 (time 1789637224 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1342 (time 1789637281.88 total 10.01 last 134.11)
 - open/close 2640 (time 1789637291.88 total 20.01 last 129.80)
 - open/close 3976 (time 1789637301.89 total 30.01 last 133.51)
 - open/close 5395 (time 1789637311.89 total 40.02 last 141.86)
 - open/close 6797 (time 1789637321.89 total 50.02 last 140.20)
 - open/close 8093 (time 1789637331.90 total 60.02 last 129.57)
 - open/close 9357 (time 1789637341.90 total 70.02 last 126.34)
total: 10000 open/close in 75.17 seconds: 133.03 ops/second
 - unlinked 0 (time 1789637349 ; total 0 ; last 0)
total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg301-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg301-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg301-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-OST0000
oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg301-client.virtnet:  -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre
On OST0, 9430 used inodes rc=0
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg301-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg301-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.101@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.101@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.101@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg301-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-OST0000
oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg301-client.virtnet:  -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg301-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg301-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server
stop mds service on oleg301-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server
unloading modules via unload_modules_local on: 'oleg301-server'
oleg301-server: oleg301-server.virtnet: executing unload_modules_local
oleg301-server: kmemleak disabled
modules unloaded.
pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2
pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2