== conf-sanity test 69: replace an OST with the same index ========================================================== 14:34:20 (1789583660)
start mds service on oleg338-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep 16 09:22 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg338-server'
oleg338-server: oleg338-server.virtnet: executing load_modules_local
oleg338-server: kmemleak disabled
oleg338-server: Loading modules from /home/green/git/lustre-release/lustre
oleg338-server: detected 4 online CPUs by sysfs
oleg338-server: MODOPTS_LIBCFS=
oleg338-server: Force libcfs to create 2 CPU partitions
oleg338-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg338-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg338-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg338-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg338-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1
Started lustre-OST0000
oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg338-client.virtnet:  -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 754 (time 1789583741.21 total 10.01 last 75.33)
 - open/close 1582 (time 1789583751.21 total 20.01 last 82.78)
 - open/close 2376 (time 1789583761.23 total 30.03 last 79.26)
 - open/close 3108 (time 1789583771.23 total 40.03 last 73.18)
 - open/close 3908 (time 1789583781.24 total 50.04 last 79.91)
 - open/close 4695 (time 1789583791.24 total 60.05 last 78.67)
 - open/close 5339 (time 1789583801.26 total 70.06 last 64.32)
 - open/close 5994 (time 1789583811.26 total 80.06 last 65.47)
 - open/close 6795 (time 1789583821.26 total 90.07 last 80.07)
 - open/close 7541 (time 1789583831.26 total 100.07 last 74.60)
 - open/close 8243 (time 1789583841.28 total 110.08 last 70.12)
 - open/close 8943 (time 1789583851.29 total 120.09 last 69.92)
 - open/close 9718 (time 1789583861.29 total 130.10 last 77.46)
total: 10000 open/close in 133.70 seconds: 74.80 ops/second
 - unlinked 0 (time 1789583868 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 805 (time 1789583964.22 total 10.01 last 80.45)
 - open/close 1565 (time 1789583974.22 total 20.01 last 75.98)
 - open/close 2391 (time 1789583984.22 total 30.01 last 82.56)
 - open/close 3174 (time 1789583994.23 total 40.02 last 78.23)
 - open/close 3869 (time 1789584004.24 total 50.03 last 69.45)
 - open/close 4675 (time 1789584014.24 total 60.03 last 80.57)
 - open/close 5464 (time 1789584024.24 total 70.04 last 78.89)
 - open/close 6277 (time 1789584034.25 total 80.04 last 81.22)
 - open/close 7096 (time 1789584044.25 total 90.04 last 81.90)
 - open/close 7925 (time 1789584054.27 total 100.06 last 82.79)
 - open/close 8732 (time 1789584064.28 total 110.07 last 80.60)
 - open/close 9516 (time 1789584074.28 total 120.07 last 78.39)
total: 10000 open/close in 126.30 seconds: 79.17 ops/second
 - unlinked 0 (time 1789584084 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 807 (time 1789584181.69 total 10.00 last 80.66)
 - open/close 1585 (time 1789584191.69 total 20.01 last 77.78)
 - open/close 2389 (time 1789584201.70 total 30.02 last 80.28)
 - open/close 3201 (time 1789584211.71 total 40.03 last 81.17)
 - open/close 3974 (time 1789584221.72 total 50.03 last 77.24)
 - open/close 4782 (time 1789584231.73 total 60.05 last 80.70)
 - open/close 5560 (time 1789584241.73 total 70.05 last 77.76)
 - open/close 6464 (time 1789584251.73 total 80.05 last 90.39)
 - open/close 7242 (time 1789584261.74 total 90.06 last 77.76)
 - open/close 8060 (time 1789584271.76 total 100.07 last 81.67)
 - open/close 8830 (time 1789584281.76 total 110.08 last 76.94)
 - open/close 9590 (time 1789584291.77 total 120.08 last 75.98)
total: 10000 open/close in 125.55 seconds: 79.65 ops/second
 - unlinked 0 (time 1789584301 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 846 (time 1789584399.79 total 10.00 last 84.56)
 - open/close 1626 (time 1789584409.79 total 20.01 last 77.98)
 - open/close 2462 (time 1789584419.80 total 30.01 last 83.55)
 - open/close 3291 (time 1789584429.80 total 40.01 last 82.90)
 - open/close 4039 (time 1789584439.80 total 50.01 last 74.79)
 - open/close 4827 (time 1789584449.80 total 60.02 last 78.76)
 - open/close 5727 (time 1789584459.80 total 70.02 last 89.98)
 - open/close 6507 (time 1789584469.81 total 80.03 last 77.96)
 - open/close 7295 (time 1789584479.81 total 90.03 last 78.79)
 - open/close 8098 (time 1789584489.82 total 100.03 last 80.27)
 - open/close 8891 (time 1789584499.83 total 110.04 last 79.22)
 - open/close 9718 (time 1789584509.83 total 120.04 last 82.68)
total: 10000 open/close in 123.56 seconds: 80.93 ops/second
 - unlinked 0 (time 1789584516 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 829 (time 1789584609.55 total 10.00 last 82.86)
 - open/close 1590 (time 1789584619.56 total 20.02 last 75.96)
 - open/close 2426 (time 1789584629.57 total 30.02 last 83.59)
 - open/close 3162 (time 1789584639.57 total 40.03 last 73.57)
 - open/close 3964 (time 1789584649.57 total 50.03 last 80.18)
 - open/close 4757 (time 1789584659.58 total 60.04 last 79.27)
 - open/close 5552 (time 1789584669.59 total 70.05 last 79.40)
 - open/close 6345 (time 1789584679.60 total 80.06 last 79.25)
 - open/close 7099 (time 1789584689.60 total 90.06 last 75.34)
 - open/close 7875 (time 1789584699.62 total 100.08 last 77.49)
 - open/close 8605 (time 1789584709.62 total 110.08 last 72.97)
 - open/close 9375 (time 1789584719.65 total 120.11 last 76.80)
total: 10000 open/close in 128.68 seconds: 77.71 ops/second
 - unlinked 0 (time 1789584732 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 830 (time 1789584825.83 total 10.01 last 82.94)
 - open/close 1650 (time 1789584835.84 total 20.02 last 81.92)
 - open/close 2403 (time 1789584845.85 total 30.02 last 75.25)
 - open/close 3250 (time 1789584855.85 total 40.02 last 84.68)
 - open/close 4006 (time 1789584865.85 total 50.02 last 75.60)
 - open/close 4768 (time 1789584875.86 total 60.04 last 76.10)
 - open/close 5616 (time 1789584885.86 total 70.04 last 84.80)
 - open/close 6446 (time 1789584895.87 total 80.04 last 82.95)
 - open/close 7259 (time 1789584905.87 total 90.05 last 81.29)
 - open/close 8069 (time 1789584915.88 total 100.05 last 80.93)
 - open/close 8874 (time 1789584925.89 total 110.06 last 80.43)
 - open/close 9692 (time 1789584935.89 total 120.07 last 81.76)
total: 10000 open/close in 124.17 seconds: 80.53 ops/second
 - unlinked 0 (time 1789584943 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 816 (time 1789585030.67 total 10.01 last 81.50)
 - open/close 1628 (time 1789585040.67 total 20.01 last 81.20)
 - open/close 2429 (time 1789585050.68 total 30.02 last 80.06)
 - open/close 3281 (time 1789585060.68 total 40.02 last 85.14)
 - open/close 4080 (time 1789585070.69 total 50.03 last 79.83)
 - open/close 4955 (time 1789585080.70 total 60.04 last 87.39)
 - open/close 5760 (time 1789585090.72 total 70.06 last 80.37)
 - open/close 6526 (time 1789585100.72 total 80.06 last 76.58)
 - open/close 7366 (time 1789585110.73 total 90.07 last 83.96)
 - open/close 8167 (time 1789585120.73 total 100.07 last 80.08)
 - open/close 9003 (time 1789585130.73 total 110.07 last 83.58)
 - open/close 9868 (time 1789585140.75 total 120.09 last 86.39)
total: 10000 open/close in 121.60 seconds: 82.24 ops/second
 - unlinked 0 (time 1789585146 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 778 (time 1789585236.81 total 10.01 last 77.73)
 - open/close 1614 (time 1789585246.81 total 20.01 last 83.55)
 - open/close 2560 (time 1789585256.82 total 30.02 last 94.53)
 - open/close 3386 (time 1789585266.82 total 40.02 last 82.59)
 - open/close 4264 (time 1789585276.83 total 50.03 last 87.71)
 - open/close 5118 (time 1789585286.85 total 60.05 last 85.23)
 - open/close 5957 (time 1789585296.86 total 70.06 last 83.84)
 - open/close 6795 (time 1789585306.86 total 80.06 last 83.79)
 - open/close 7647 (time 1789585316.86 total 90.06 last 85.20)
 - open/close 8425 (time 1789585326.87 total 100.07 last 77.76)
 - open/close 9246 (time 1789585336.87 total 110.07 last 82.06)
total: 10000 open/close in 119.85 seconds: 83.44 ops/second
 - unlinked 0 (time 1789585350 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 811 (time 1789585444.88 total 10.01 last 81.01)
 - open/close 1643 (time 1789585454.90 total 20.03 last 83.01)
 - open/close 2285 (time 1789585464.95 total 30.08 last 63.89)
 - open/close 3084 (time 1789585474.96 total 40.09 last 79.83)
 - open/close 3886 (time 1789585484.97 total 50.11 last 80.09)
 - open/close 4706 (time 1789585494.98 total 60.11 last 81.99)
 - open/close 5580 (time 1789585504.98 total 70.11 last 87.37)
 - open/close 6531 (time 1789585514.98 total 80.12 last 95.05)
 - open/close 7430 (time 1789585525.01 total 90.14 last 89.69)
 - open/close 8303 (time 1789585535.02 total 100.15 last 87.21)
 - open/close 9110 (time 1789585545.03 total 110.16 last 80.63)
 - open/close 9898 (time 1789585555.03 total 120.16 last 78.78)
total: 10000 open/close in 121.46 seconds: 82.33 ops/second
 - unlinked 0 (time 1789585560 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 840 (time 1789585647.13 total 10.02 last 83.84)
 - open/close 1649 (time 1789585657.13 total 20.02 last 80.87)
 - open/close 2453 (time 1789585667.14 total 30.02 last 80.40)
 - open/close 3303 (time 1789585677.14 total 40.03 last 84.92)
 - open/close 4086 (time 1789585687.15 total 50.04 last 78.27)
 - open/close 4848 (time 1789585697.15 total 60.04 last 76.15)
 - open/close 5658 (time 1789585707.16 total 70.05 last 80.96)
 - open/close 6420 (time 1789585717.17 total 80.06 last 76.10)
 - open/close 7174 (time 1789585727.19 total 90.07 last 75.30)
 - open/close 8022 (time 1789585737.20 total 100.08 last 84.72)
 - open/close 8851 (time 1789585747.20 total 110.09 last 82.88)
 - open/close 9626 (time 1789585757.20 total 120.09 last 77.45)
total: 10000 open/close in 125.44 seconds: 79.72 ops/second
 - unlinked 0 (time 1789585766 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg338-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg338-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg338-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1
Started lustre-OST0000
oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg338-server: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg338-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg338-client.virtnet:  -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg338-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg338-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.138@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.138@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.138@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.138@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg338-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1
Started lustre-OST0000
oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg338-server: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg338-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg338-client.virtnet:  -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg338-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg338-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server
stop mds service on oleg338-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server
unloading modules via unload_modules_local on: 'oleg338-server'
oleg338-server: oleg338-server.virtnet: executing unload_modules_local
oleg338-server: kmemleak disabled
modules unloaded.
pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2
pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2
pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2