== conf-sanity test 69: replace an OST with the same index ========================================================== 02:23:37 (1789367017)
start mds service on oleg160-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep 14 01:55 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: kmemleak disabled
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: MODOPTS_LIBCFS=
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg160-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 757 (time 1789367096.68 total 10.00 last 75.69)
 - open/close 1635 (time 1789367106.68 total 20.01 last 87.76)
 - open/close 2432 (time 1789367116.68 total 30.01 last 79.69)
 - open/close 3316 (time 1789367126.69 total 40.01 last 88.37)
 - open/close 4234 (time 1789367136.69 total 50.02 last 91.74)
 - open/close 5147 (time 1789367146.70 total 60.02 last 91.29)
 - open/close 6096 (time 1789367156.71 total 70.03 last 94.80)
 - open/close 7142 (time 1789367166.72 total 80.04 last 104.50)
 - open/close 8121 (time 1789367176.73 total 90.05 last 97.79)
 - open/close 9115 (time 1789367186.73 total 100.06 last 99.34)
 - open/close 9737 (time 1789367196.74 total 110.06 last 62.17)
total: 10000 open/close in 113.66 seconds: 87.98 ops/second
 - unlinked 0 (time 1789367203 ; total 0 ; last 0)
total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second
 - open/close 1085 (time 1789367278.07 total 10.00 last 108.46)
 - open/close 1967 (time 1789367288.07 total 20.00 last 88.18)
 - open/close 2784 (time 1789367298.08 total 30.02 last 81.61)
 - open/close 3845 (time 1789367308.08 total 40.02 last 106.06)
 - open/close 4923 (time 1789367318.09 total 50.03 last 107.74)
 - open/close 6089 (time 1789367328.10 total 60.04 last 116.48)
 - open/close 7199 (time 1789367338.11 total 70.04 last 110.92)
 - open/close 8293 (time 1789367348.11 total 80.05 last 109.34)
 - open/close 9276 (time 1789367358.12 total 90.06 last 98.18)
total: 10000 open/close in 99.19 seconds: 100.81 ops/second
 - unlinked 0 (time 1789367371 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 1038 (time 1789367442.81 total 10.01 last 103.74)
 - open/close 2047 (time 1789367452.82 total 20.02 last 100.79)
 - open/close 3017 (time 1789367462.83 total 30.03 last 96.91)
 - open/close 3825 (time 1789367472.83 total 40.03 last 80.79)
 - open/close 4847 (time 1789367482.83 total 50.03 last 102.16)
 - open/close 5808 (time 1789367492.85 total 60.04 last 95.99)
 - open/close 6727 (time 1789367502.85 total 70.05 last 91.84)
 - open/close 7581 (time 1789367512.86 total 80.05 last 85.36)
 - open/close 8553 (time 1789367522.87 total 90.06 last 97.10)
 - open/close 9452 (time 1789367532.88 total 100.08 last 89.80)
total: 10000 open/close in 107.54 seconds: 92.99 ops/second
 - unlinked 0 (time 1789367544 ; total 0 ; last 0)
total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second
 - open/close 1192 (time 1789367620.74 total 10.00 last 119.14)
 - open/close 2476 (time 1789367630.74 total 20.01 last 128.36)
 - open/close 3628 (time 1789367640.75 total 30.01 last 115.16)
 - open/close 4750 (time 1789367650.75 total 40.02 last 112.14)
 - open/close 5825 (time 1789367660.75 total 50.02 last 107.48)
 - open/close 6928 (time 1789367670.76 total 60.02 last 110.26)
 - open/close 8153 (time 1789367680.76 total 70.02 last 122.47)
 - open/close 9242 (time 1789367690.77 total 80.03 last 108.84)
total: 10000 open/close in 87.37 seconds: 114.45 ops/second
 - unlinked 0 (time 1789367700 ; total 0 ; last 0)
total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second
 - open/close 1148 (time 1789367771.32 total 10.00 last 114.77)
 - open/close 2187 (time 1789367781.33 total 20.01 last 103.88)
 - open/close 3295 (time 1789367791.33 total 30.01 last 110.77)
 - open/close 4490 (time 1789367801.33 total 40.01 last 119.43)
 - open/close 5611 (time 1789367811.34 total 50.02 last 112.04)
 - open/close 6773 (time 1789367821.34 total 60.02 last 116.15)
 - open/close 8005 (time 1789367831.35 total 70.02 last 123.18)
 - open/close 9267 (time 1789367841.35 total 80.03 last 126.13)
total: 10000 open/close in 86.01 seconds: 116.26 ops/second
 - unlinked 0 (time 1789367849 ; total 0 ; last 0)
total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second
 - open/close 1187 (time 1789367908.46 total 10.00 last 118.68)
 - open/close 2496 (time 1789367918.46 total 20.00 last 130.87)
 - open/close 3877 (time 1789367928.46 total 30.01 last 138.07)
 - open/close 5359 (time 1789367938.47 total 40.02 last 148.07)
 - open/close 6792 (time 1789367948.47 total 50.02 last 143.29)
 - open/close 8156 (time 1789367958.48 total 60.02 last 136.34)
 - open/close 9383 (time 1789367968.48 total 70.03 last 122.65)
total: 10000 open/close in 74.35 seconds: 134.50 ops/second
 - unlinked 0 (time 1789367974 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1229 (time 1789368031.99 total 10.00 last 122.89)
 - open/close 2517 (time 1789368041.99 total 20.00 last 128.76)
 - open/close 3799 (time 1789368051.99 total 30.01 last 128.18)
 - open/close 5100 (time 1789368062.00 total 40.01 last 129.99)
 - open/close 6313 (time 1789368072.01 total 50.02 last 121.24)
 - open/close 7495 (time 1789368082.01 total 60.03 last 118.12)
 - open/close 8621 (time 1789368092.02 total 70.03 last 112.54)
 - open/close 9798 (time 1789368102.02 total 80.04 last 117.64)
total: 10000 open/close in 81.87 seconds: 122.14 ops/second
 - unlinked 0 (time 1789368107 ; total 1 ; last 1)
total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second
 - open/close 1301 (time 1789368166.27 total 10.00 last 130.09)
 - open/close 2638 (time 1789368176.27 total 20.00 last 133.68)
 - open/close 3976 (time 1789368186.28 total 30.01 last 133.72)
 - open/close 5189 (time 1789368196.29 total 40.02 last 121.19)
 - open/close 6529 (time 1789368206.29 total 50.02 last 133.96)
 - open/close 7819 (time 1789368216.29 total 60.02 last 128.98)
 - open/close 9072 (time 1789368226.29 total 70.02 last 125.29)
total: 10000 open/close in 77.43 seconds: 129.15 ops/second
 - unlinked 0 (time 1789368236 ; total 0 ; last 0)
total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second
 - open/close 1324 (time 1789368292.30 total 10.00 last 132.40)
 - open/close 2762 (time 1789368302.30 total 20.00 last 143.74)
 - open/close 4123 (time 1789368312.30 total 30.01 last 136.08)
 - open/close 5417 (time 1789368322.31 total 40.01 last 129.34)
 - open/close 6629 (time 1789368332.31 total 50.01 last 121.20)
 - open/close 7899 (time 1789368342.31 total 60.01 last 126.96)
 - open/close 9405 (time 1789368352.32 total 70.03 last 150.38)
total: 10000 open/close in 74.15 seconds: 134.85 ops/second
 - unlinked 0 (time 1789368358 ; total 0 ; last 0)
total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second
 - open/close 1281 (time 1789368415.10 total 10.01 last 128.03)
 - open/close 2643 (time 1789368425.11 total 20.01 last 136.14)
 - open/close 3882 (time 1789368435.11 total 30.02 last 123.78)
 - open/close 5251 (time 1789368445.11 total 40.02 last 136.90)
 - open/close 6598 (time 1789368455.12 total 50.02 last 134.65)
 - open/close 7994 (time 1789368465.13 total 60.03 last 139.50)
 - open/close 9438 (time 1789368475.13 total 70.03 last 144.40)
total: 10000 open/close in 74.72 seconds: 133.84 ops/second
 - unlinked 0 (time 1789368482 ; total 0 ; last 0)
total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
On OST0, 19446 used inodes rc=0
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.160@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.160@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: kmemleak disabled
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2