== conf-sanity test 69: replace an OST with the same index ========================================================== 20:18:16 (1789517896)
start mds service on oleg636-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep 15 17:59 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg636-server'
oleg636-server: oleg636-server.virtnet: executing load_modules_local
oleg636-server: kmemleak disabled
oleg636-server: Loading modules from /home/green/git/lustre-release/lustre
oleg636-server: detected 4 online CPUs by sysfs
oleg636-server: MODOPTS_LIBCFS=
oleg636-server: Force libcfs to create 2 CPU partitions
oleg636-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg636-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg636-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg636-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1075 (time 1789517949.88 total 10.00 last 107.47)
 - open/close 2027 (time 1789517959.88 total 20.01 last 95.13)
 - open/close 3004 (time 1789517969.89 total 30.02 last 97.64)
 - open/close 4011 (time 1789517979.89 total 40.02 last 100.68)
 - open/close 5020 (time 1789517989.90 total 50.03 last 100.79)
 - open/close 5992 (time 1789517999.90 total 60.03 last 97.20)
 - open/close 6962 (time 1789518009.91 total 70.03 last 96.95)
 - open/close 7974 (time 1789518019.91 total 80.04 last 101.16)
 - open/close 8993 (time 1789518029.91 total 90.04 last 101.88)
total: 10000 open/close in 99.56 seconds: 100.45 ops/second
 - unlinked 0 (time 1789518041 ; total 0 ; last 0)
total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second
 - open/close 1069 (time 1789518106.78 total 10.00 last 106.88)
 - open/close 2096 (time 1789518116.79 total 20.01 last 102.64)
 - open/close 3229 (time 1789518126.79 total 30.01 last 113.29)
 - open/close 4359 (time 1789518136.80 total 40.02 last 112.92)
 - open/close 5323 (time 1789518146.80 total 50.02 last 96.37)
 - open/close 6511 (time 1789518156.80 total 60.02 last 118.77)
 - open/close 7678 (time 1789518166.81 total 70.03 last 116.65)
 - open/close 8884 (time 1789518176.81 total 80.03 last 120.57)
 - open/close 9962 (time 1789518186.81 total 90.03 last 107.75)
total: 10000 open/close in 90.37 seconds: 110.65 ops/second
 - unlinked 0 (time 1789518189 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1193 (time 1789518252.60 total 10.00 last 119.29)
 - open/close 2149 (time 1789518262.60 total 20.00 last 95.59)
 - open/close 3275 (time 1789518272.61 total 30.00 last 112.60)
 - open/close 4440 (time 1789518282.61 total 40.01 last 116.44)
 - open/close 5589 (time 1789518292.61 total 50.01 last 114.87)
 - open/close 6873 (time 1789518302.62 total 60.01 last 128.34)
 - open/close 8140 (time 1789518312.62 total 70.02 last 126.65)
 - open/close 9433 (time 1789518322.62 total 80.02 last 129.28)
total: 10000 open/close in 84.27 seconds: 118.67 ops/second
 - unlinked 0 (time 1789518329 ; total 0 ; last 0)
total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second
 - open/close 1144 (time 1789518388.68 total 10.00 last 114.40)
 - open/close 2521 (time 1789518398.68 total 20.00 last 137.69)
 - open/close 3863 (time 1789518408.69 total 30.01 last 134.14)
 - open/close 5070 (time 1789518418.69 total 40.01 last 120.69)
 - open/close 6270 (time 1789518428.70 total 50.01 last 119.94)
 - open/close 7503 (time 1789518438.71 total 60.02 last 123.16)
 - open/close 8672 (time 1789518448.71 total 70.03 last 116.82)
 - open/close 9858 (time 1789518458.72 total 80.04 last 118.50)
total: 10000 open/close in 81.11 seconds: 123.29 ops/second
 - unlinked 0 (time 1789518462 ; total 1 ; last 1)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 1020 (time 1789518525.09 total 10.01 last 101.93)
 - open/close 2225 (time 1789518535.10 total 20.01 last 120.43)
 - open/close 3426 (time 1789518545.10 total 30.02 last 120.02)
 - open/close 4529 (time 1789518555.11 total 40.02 last 110.25)
 - open/close 5551 (time 1789518565.11 total 50.03 last 102.16)
 - open/close 6906 (time 1789518575.12 total 60.03 last 135.44)
 - open/close 8087 (time 1789518585.12 total 70.04 last 118.04)
 - open/close 9235 (time 1789518595.12 total 80.04 last 114.76)
total: 10000 open/close in 86.71 seconds: 115.33 ops/second
 - unlinked 0 (time 1789518604 ; total 0 ; last 0)
total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second
 - open/close 1137 (time 1789518664.61 total 10.01 last 113.63)
 - open/close 2240 (time 1789518674.61 total 20.01 last 110.25)
 - open/close 3392 (time 1789518684.62 total 30.02 last 115.13)
 - open/close 4665 (time 1789518694.62 total 40.02 last 127.29)
 - open/close 5992 (time 1789518704.62 total 50.02 last 132.65)
 - open/close 7170 (time 1789518714.63 total 60.02 last 117.78)
 - open/close 8483 (time 1789518724.63 total 70.02 last 131.28)
 - open/close 9788 (time 1789518734.63 total 80.03 last 130.42)
total: 10000 open/close in 81.41 seconds: 122.84 ops/second
 - unlinked 0 (time 1789518738 ; total 0 ; last 0)
total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second
 - open/close 1092 (time 1789518798.48 total 10.00 last 109.20)
 - open/close 2179 (time 1789518808.48 total 20.00 last 108.68)
 - open/close 3245 (time 1789518818.49 total 30.01 last 106.53)
 - open/close 4335 (time 1789518828.49 total 40.01 last 108.95)
 - open/close 5413 (time 1789518838.49 total 50.01 last 107.79)
 - open/close 6542 (time 1789518848.50 total 60.02 last 112.81)
 - open/close 7663 (time 1789518858.50 total 70.03 last 112.05)
 - open/close 8809 (time 1789518868.51 total 80.03 last 114.56)
 - open/close 9926 (time 1789518878.51 total 90.03 last 111.68)
total: 10000 open/close in 90.72 seconds: 110.22 ops/second
 - unlinked 0 (time 1789518881 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1220 (time 1789518944.41 total 10.00 last 121.99)
 - open/close 2337 (time 1789518954.41 total 20.01 last 111.64)
 - open/close 3404 (time 1789518964.42 total 30.01 last 106.62)
 - open/close 4484 (time 1789518974.42 total 40.02 last 107.94)
 - open/close 5633 (time 1789518984.43 total 50.03 last 114.77)
 - open/close 6734 (time 1789518994.44 total 60.04 last 110.02)
 - open/close 7934 (time 1789519004.44 total 70.04 last 119.97)
 - open/close 9097 (time 1789519014.46 total 80.05 last 116.13)
total: 10000 open/close in 87.43 seconds: 114.37 ops/second
 - unlinked 0 (time 1789519024 ; total 0 ; last 0)
total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second
 - open/close 1212 (time 1789519088.76 total 10.00 last 121.20)
 - open/close 2406 (time 1789519098.76 total 20.00 last 119.36)
 - open/close 3633 (time 1789519108.77 total 30.02 last 122.55)
 - open/close 4876 (time 1789519118.77 total 40.02 last 124.27)
 - open/close 5968 (time 1789519128.78 total 50.02 last 109.14)
 - open/close 7010 (time 1789519138.78 total 60.03 last 104.14)
 - open/close 8048 (time 1789519148.79 total 70.03 last 103.76)
 - open/close 9234 (time 1789519158.81 total 80.06 last 118.29)
total: 10000 open/close in 86.75 seconds: 115.27 ops/second
 - unlinked 0 (time 1789519167 ; total 0 ; last 0)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 979 (time 1789519231.97 total 10.00 last 97.86)
 - open/close 2015 (time 1789519241.97 total 20.01 last 103.56)
 - open/close 3095 (time 1789519251.98 total 30.01 last 107.95)
 - open/close 4193 (time 1789519261.98 total 40.02 last 109.75)
 - open/close 5248 (time 1789519271.99 total 50.02 last 105.45)
 - open/close 6326 (time 1789519281.99 total 60.02 last 107.78)
 - open/close 7549 (time 1789519292.00 total 70.03 last 122.18)
 - open/close 8603 (time 1789519302.00 total 80.04 last 105.33)
 - open/close 9563 (time 1789519312.01 total 90.05 last 95.94)
total: 10000 open/close in 94.04 seconds: 106.34 ops/second
 - unlinked 0 (time 1789519318 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg636-server: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg636-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
On OST0, 19422 used inodes rc=0
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.136@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg636-server: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg636-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server
stop mds service on oleg636-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server
unloading modules via unload_modules_local on: 'oleg636-server'
oleg636-server: oleg636-server.virtnet: executing unload_modules_local
oleg636-server: kmemleak disabled
modules unloaded.
pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2