== conf-sanity test 69: replace an OST with the same index ========================================================== 01:38:33 (1788586713)
start mds service on oleg254-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921056 Sep  4 09:57 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg254-server'
oleg254-server: oleg254-server.virtnet: executing load_modules_local
oleg254-server: kmemleak disabled
oleg254-server: Loading modules from /home/green/git/lustre-release/lustre
oleg254-server: detected 4 online CPUs by sysfs
oleg254-server: MODOPTS_LIBCFS=
oleg254-server: Force libcfs to create 2 CPU partitions
oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg254-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg254-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg254-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-OST0000
oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg254-client.virtnet:  -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 999 (time 1788586795.18 total 10.00 last 99.87)
 - open/close 1872 (time 1788586805.19 total 20.01 last 87.28)
 - open/close 2876 (time 1788586815.19 total 30.01 last 100.36)
 - open/close 3814 (time 1788586825.20 total 40.02 last 93.74)
 - open/close 4733 (time 1788586835.20 total 50.02 last 91.89)
 - open/close 5546 (time 1788586845.21 total 60.03 last 81.20)
 - open/close 6448 (time 1788586855.21 total 70.03 last 90.16)
 - open/close 7296 (time 1788586865.22 total 80.03 last 84.79)
 - open/close 8075 (time 1788586875.22 total 90.04 last 77.84)
 - open/close 8903 (time 1788586885.22 total 100.04 last 82.80)
 - open/close 9808 (time 1788586895.23 total 110.05 last 90.47)
total: 10000 open/close in 112.13 seconds: 89.18 ops/second
 - unlinked 0 (time 1788586901 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 1025 (time 1788586989.30 total 10.00 last 102.47)
 - open/close 2042 (time 1788586999.31 total 20.01 last 101.63)
 - open/close 2912 (time 1788587009.32 total 30.02 last 86.91)
 - open/close 3861 (time 1788587019.33 total 40.03 last 94.81)
 - open/close 4729 (time 1788587029.33 total 50.03 last 86.77)
 - open/close 5689 (time 1788587039.34 total 60.04 last 95.96)
 - open/close 6461 (time 1788587049.35 total 70.05 last 77.11)
 - open/close 7229 (time 1788587059.36 total 80.06 last 76.71)
 - open/close 8007 (time 1788587069.36 total 90.06 last 77.77)
 - open/close 8741 (time 1788587079.37 total 100.07 last 73.35)
 - open/close 9431 (time 1788587089.38 total 110.08 last 68.95)
total: 10000 open/close in 118.51 seconds: 84.38 ops/second
 - unlinked 0 (time 1788587102 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 775 (time 1788587190.38 total 10.01 last 77.44)
 - open/close 1567 (time 1788587200.39 total 20.02 last 79.12)
 - open/close 2376 (time 1788587210.40 total 30.02 last 80.88)
 - open/close 3215 (time 1788587220.40 total 40.03 last 83.84)
 - open/close 4034 (time 1788587230.43 total 50.05 last 81.71)
 - open/close 4849 (time 1788587240.43 total 60.05 last 81.47)
 - open/close 5523 (time 1788587250.43 total 70.06 last 67.39)
 - open/close 6411 (time 1788587260.44 total 80.06 last 88.75)
 - open/close 7424 (time 1788587270.45 total 90.07 last 101.22)
 - open/close 8430 (time 1788587280.45 total 100.07 last 100.57)
 - open/close 9437 (time 1788587290.45 total 110.07 last 100.68)
total: 10000 open/close in 116.97 seconds: 85.49 ops/second
 - unlinked 0 (time 1788587302 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 1012 (time 1788587386.28 total 10.00 last 101.16)
 - open/close 1914 (time 1788587396.28 total 20.01 last 90.16)
 - open/close 2813 (time 1788587406.30 total 30.03 last 89.74)
 - open/close 3515 (time 1788587416.31 total 40.04 last 70.14)
 - open/close 4269 (time 1788587426.31 total 50.04 last 75.36)
 - open/close 5050 (time 1788587436.31 total 60.04 last 78.08)
 - open/close 5830 (time 1788587446.32 total 70.05 last 77.94)
 - open/close 6669 (time 1788587456.33 total 80.05 last 83.86)
 - open/close 7502 (time 1788587466.34 total 90.06 last 83.22)
 - open/close 8418 (time 1788587476.34 total 100.07 last 91.57)
 - open/close 9305 (time 1788587486.35 total 110.08 last 88.60)
total: 10000 open/close in 117.75 seconds: 84.93 ops/second
 - unlinked 0 (time 1788587498 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 873 (time 1788587589.59 total 10.01 last 87.23)
 - open/close 1717 (time 1788587599.59 total 20.02 last 84.33)
 - open/close 2599 (time 1788587609.59 total 30.02 last 88.19)
 - open/close 3462 (time 1788587619.60 total 40.03 last 86.23)
 - open/close 4330 (time 1788587629.61 total 50.03 last 86.75)
 - open/close 5111 (time 1788587639.62 total 60.04 last 78.00)
 - open/close 5818 (time 1788587649.63 total 70.05 last 70.66)
 - open/close 6533 (time 1788587659.64 total 80.07 last 71.38)
 - open/close 7313 (time 1788587669.65 total 90.07 last 77.96)
 - open/close 7996 (time 1788587679.66 total 100.08 last 68.24)
 - open/close 8871 (time 1788587689.66 total 110.08 last 87.46)
 - open/close 9718 (time 1788587699.67 total 120.09 last 84.63)
total: 10000 open/close in 123.17 seconds: 81.19 ops/second
 - unlinked 0 (time 1788587707 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 868 (time 1788587797.17 total 10.01 last 86.76)
 - open/close 1706 (time 1788587807.18 total 20.02 last 83.71)
 - open/close 2643 (time 1788587817.18 total 30.02 last 93.69)
 - open/close 3534 (time 1788587827.18 total 40.02 last 89.09)
 - open/close 4414 (time 1788587837.18 total 50.02 last 87.98)
 - open/close 5292 (time 1788587847.19 total 60.03 last 87.73)
 - open/close 6122 (time 1788587857.20 total 70.03 last 82.95)
 - open/close 7013 (time 1788587867.20 total 80.03 last 89.10)
 - open/close 7870 (time 1788587877.20 total 90.04 last 85.69)
 - open/close 8803 (time 1788587887.21 total 100.04 last 93.23)
 - open/close 9626 (time 1788587897.21 total 110.04 last 82.29)
total: 10000 open/close in 113.90 seconds: 87.80 ops/second
 - unlinked 0 (time 1788587905 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 856 (time 1788587995.98 total 10.01 last 85.52)
 - open/close 1829 (time 1788588005.98 total 20.01 last 97.29)
 - open/close 2793 (time 1788588015.99 total 30.01 last 96.37)
 - open/close 3614 (time 1788588025.99 total 40.02 last 82.07)
 - open/close 4575 (time 1788588036.00 total 50.03 last 96.02)
 - open/close 5485 (time 1788588046.01 total 60.03 last 90.92)
 - open/close 6126 (time 1788588056.03 total 70.06 last 63.95)
 - open/close 7131 (time 1788588066.06 total 80.09 last 100.19)
 - open/close 7868 (time 1788588076.07 total 90.09 last 73.68)
 - open/close 8605 (time 1788588086.07 total 100.09 last 73.70)
 - open/close 9416 (time 1788588096.07 total 110.09 last 81.09)
total: 10000 open/close in 116.79 seconds: 85.62 ops/second
 - unlinked 0 (time 1788588107 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 755 (time 1788588203.19 total 10.00 last 75.49)
 - open/close 1551 (time 1788588213.20 total 20.00 last 79.58)
 - open/close 2401 (time 1788588223.20 total 30.01 last 84.96)
 - open/close 3241 (time 1788588233.21 total 40.02 last 83.91)
 - open/close 4079 (time 1788588243.22 total 50.02 last 83.76)
 - open/close 4860 (time 1788588253.23 total 60.04 last 77.97)
 - open/close 5700 (time 1788588263.24 total 70.05 last 83.92)
 - open/close 6599 (time 1788588273.24 total 80.05 last 89.88)
 - open/close 7534 (time 1788588283.25 total 90.06 last 93.43)
 - open/close 8370 (time 1788588293.26 total 100.07 last 83.52)
 - open/close 9188 (time 1788588303.26 total 110.07 last 81.79)
total: 10000 open/close in 119.75 seconds: 83.51 ops/second
 - unlinked 0 (time 1788588317 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 959 (time 1788588403.84 total 10.00 last 95.87)
 - open/close 1783 (time 1788588413.84 total 20.00 last 82.39)
 - open/close 2616 (time 1788588423.84 total 30.01 last 83.28)
 - open/close 3508 (time 1788588433.85 total 40.01 last 89.13)
 - open/close 4398 (time 1788588443.85 total 50.02 last 88.97)
 - open/close 5184 (time 1788588453.86 total 60.03 last 78.53)
 - open/close 6024 (time 1788588463.87 total 70.04 last 83.91)
 - open/close 6815 (time 1788588473.87 total 80.04 last 79.08)
 - open/close 7527 (time 1788588483.88 total 90.05 last 71.15)
 - open/close 8296 (time 1788588493.88 total 100.05 last 76.88)
 - open/close 9213 (time 1788588503.90 total 110.06 last 91.60)
total: 10000 open/close in 119.76 seconds: 83.50 ops/second
 - unlinked 0 (time 1788588517 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 831 (time 1788588613.34 total 10.02 last 82.96)
 - open/close 1720 (time 1788588623.34 total 20.02 last 88.90)
 - open/close 2651 (time 1788588633.35 total 30.02 last 93.04)
 - open/close 3580 (time 1788588643.36 total 40.03 last 92.80)
 - open/close 4388 (time 1788588653.38 total 50.05 last 80.63)
 - open/close 5156 (time 1788588663.39 total 60.06 last 76.72)
 - open/close 5865 (time 1788588673.41 total 70.08 last 70.80)
 - open/close 6669 (time 1788588683.41 total 80.08 last 80.36)
 - open/close 7116 (time 1788588693.42 total 90.09 last 44.65)
 - open/close 7735 (time 1788588703.43 total 100.11 last 61.82)
 - open/close 8578 (time 1788588713.43 total 110.11 last 84.29)
 - open/close 9285 (time 1788588723.44 total 120.11 last 70.68)
total: 10000 open/close in 128.82 seconds: 77.63 ops/second
 - unlinked 0 (time 1788588736 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg254-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg254-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.154@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg254-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-OST0000
oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg254-client.virtnet:  -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg254-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg254-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.154@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.154@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.154@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.154@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg254-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-OST0000
oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg254-client.virtnet:  -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg254-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg254-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server
stop mds service on oleg254-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server
unloading modules via unload_modules_local on: 'oleg254-server'
oleg254-server: oleg254-server.virtnet: executing unload_modules_local
oleg254-server: kmemleak disabled
modules unloaded.
pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2
pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2