== conf-sanity test 69: replace an OST with the same index ========================================================== 16:38:06 (1788813486)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931936 Sep  7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 723 (time 1788813590.26 total 10.00 last 72.27)
 - open/close 1435 (time 1788813600.27 total 20.01 last 71.15)
 - open/close 2124 (time 1788813610.28 total 30.02 last 68.84)
 - open/close 2856 (time 1788813620.28 total 40.02 last 73.20)
 - open/close 3567 (time 1788813630.29 total 50.03 last 71.02)
 - open/close 4291 (time 1788813640.29 total 60.03 last 72.39)
 - open/close 5027 (time 1788813650.29 total 70.03 last 73.59)
 - open/close 5673 (time 1788813660.30 total 80.04 last 64.54)
 - open/close 6356 (time 1788813670.31 total 90.06 last 68.21)
 - open/close 7112 (time 1788813680.32 total 100.06 last 75.54)
 - open/close 7776 (time 1788813690.32 total 110.07 last 66.39)
 - open/close 8524 (time 1788813700.34 total 120.08 last 74.70)
 - open/close 9245 (time 1788813710.35 total 130.09 last 72.02)
 - open/close 9899 (time 1788813720.35 total 140.10 last 65.37)
total: 10000 open/close in 141.93 seconds: 70.46 ops/second
 - unlinked 0 (time 1788813726 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 674 (time 1788813830.92 total 10.00 last 67.38)
 - open/close 1350 (time 1788813840.92 total 20.00 last 67.60)
 - open/close 2074 (time 1788813850.94 total 30.03 last 72.22)
 - open/close 2758 (time 1788813860.96 total 40.05 last 68.28)
 - open/close 3444 (time 1788813871.00 total 50.08 last 68.35)
 - open/close 4117 (time 1788813881.01 total 60.09 last 67.24)
 - open/close 4875 (time 1788813891.03 total 70.11 last 75.63)
 - open/close 5523 (time 1788813901.03 total 80.12 last 64.78)
 - open/close 6299 (time 1788813911.04 total 90.13 last 77.52)
 - open/close 7051 (time 1788813921.05 total 100.13 last 75.15)
 - open/close 7706 (time 1788813931.06 total 110.14 last 65.47)
 - open/close 8439 (time 1788813941.07 total 120.15 last 73.19)
 - open/close 9147 (time 1788813951.08 total 130.16 last 70.76)
 - open/close 9858 (time 1788813961.09 total 140.17 last 71.04)
total: 10000 open/close in 142.04 seconds: 70.40 ops/second
 - unlinked 0 (time 1788813967 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 626 (time 1788814069.75 total 10.01 last 62.54)
 - open/close 1266 (time 1788814079.75 total 20.02 last 63.96)
 - open/close 1899 (time 1788814089.76 total 30.03 last 63.22)
 - open/close 2546 (time 1788814099.77 total 40.04 last 64.64)
 - open/close 3228 (time 1788814109.78 total 50.04 last 68.17)
 - open/close 3864 (time 1788814119.78 total 60.05 last 63.57)
 - open/close 4515 (time 1788814129.78 total 70.05 last 65.09)
 - open/close 5140 (time 1788814139.81 total 80.07 last 62.33)
 - open/close 5922 (time 1788814149.82 total 90.08 last 78.13)
 - open/close 6683 (time 1788814159.86 total 100.13 last 75.78)
 - open/close 7446 (time 1788814169.87 total 110.13 last 76.25)
 - open/close 8137 (time 1788814179.88 total 120.14 last 69.01)
 - open/close 8885 (time 1788814189.88 total 130.15 last 74.78)
 - open/close 9593 (time 1788814199.89 total 140.15 last 70.79)
total: 10000 open/close in 145.62 seconds: 68.67 ops/second
 - unlinked 0 (time 1788814209 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 777 (time 1788814303.97 total 10.01 last 77.66)
 - open/close 1534 (time 1788814313.98 total 20.01 last 75.64)
 - open/close 2372 (time 1788814323.98 total 30.02 last 83.78)
 - open/close 3135 (time 1788814333.99 total 40.02 last 76.26)
 - open/close 3894 (time 1788814344.01 total 50.05 last 75.70)
 - open/close 4704 (time 1788814354.02 total 60.05 last 80.97)
 - open/close 5615 (time 1788814364.02 total 70.05 last 91.08)
 - open/close 6416 (time 1788814374.04 total 80.07 last 79.98)
 - open/close 7215 (time 1788814384.05 total 90.08 last 79.82)
 - open/close 8005 (time 1788814394.05 total 100.08 last 78.96)
 - open/close 8819 (time 1788814404.05 total 110.09 last 81.38)
 - open/close 9611 (time 1788814414.06 total 120.09 last 79.16)
total: 10000 open/close in 124.89 seconds: 80.07 ops/second
 - unlinked 0 (time 1788814422 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 727 (time 1788814515.92 total 10.01 last 72.63)
 - open/close 1546 (time 1788814525.93 total 20.02 last 81.84)
 - open/close 2384 (time 1788814535.94 total 30.03 last 83.73)
 - open/close 3203 (time 1788814545.95 total 40.04 last 81.77)
 - open/close 4008 (time 1788814555.96 total 50.04 last 80.47)
 - open/close 4868 (time 1788814565.97 total 60.06 last 85.86)
 - open/close 5680 (time 1788814575.98 total 70.07 last 81.15)
 - open/close 6496 (time 1788814585.98 total 80.07 last 81.56)
 - open/close 7298 (time 1788814595.98 total 90.07 last 80.20)
 - open/close 8057 (time 1788814605.99 total 100.08 last 75.85)
 - open/close 8876 (time 1788814615.99 total 110.08 last 81.89)
 - open/close 9750 (time 1788814625.99 total 120.08 last 87.36)
total: 10000 open/close in 122.96 seconds: 81.33 ops/second
 - unlinked 0 (time 1788814632 ; total 1 ; last 1)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 840 (time 1788814712.99 total 10.01 last 83.93)
 - open/close 1733 (time 1788814722.99 total 20.01 last 89.25)
 - open/close 2530 (time 1788814733.00 total 30.02 last 79.64)
 - open/close 3391 (time 1788814743.00 total 40.02 last 86.08)
 - open/close 4304 (time 1788814753.00 total 50.03 last 91.29)
 - open/close 5163 (time 1788814763.01 total 60.03 last 85.85)
 - open/close 5964 (time 1788814773.02 total 70.04 last 80.06)
 - open/close 6822 (time 1788814783.02 total 80.04 last 85.75)
 - open/close 7690 (time 1788814793.02 total 90.05 last 86.77)
 - open/close 8577 (time 1788814803.02 total 100.05 last 88.70)
 - open/close 9481 (time 1788814813.02 total 110.05 last 90.40)
total: 10000 open/close in 115.97 seconds: 86.23 ops/second
 - unlinked 0 (time 1788814822 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 820 (time 1788814910.89 total 10.00 last 82.00)
 - open/close 1622 (time 1788814920.90 total 20.01 last 80.15)
 - open/close 2460 (time 1788814930.90 total 30.01 last 83.78)
 - open/close 3305 (time 1788814940.90 total 40.01 last 84.48)
 - open/close 4147 (time 1788814950.91 total 50.02 last 84.12)
 - open/close 4980 (time 1788814960.92 total 60.03 last 83.26)
 - open/close 5802 (time 1788814970.92 total 70.03 last 82.17)
 - open/close 6629 (time 1788814980.92 total 80.03 last 82.68)
 - open/close 7433 (time 1788814990.93 total 90.04 last 80.36)
 - open/close 8271 (time 1788815000.93 total 100.04 last 83.74)
 - open/close 9125 (time 1788815010.95 total 110.05 last 85.30)
 - open/close 9967 (time 1788815020.95 total 120.06 last 84.15)
total: 10000 open/close in 120.44 seconds: 83.03 ops/second
 - unlinked 0 (time 1788815024 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 840 (time 1788815106.69 total 10.00 last 84.00)
 - open/close 1707 (time 1788815116.69 total 20.01 last 86.63)
 - open/close 2571 (time 1788815126.70 total 30.02 last 86.34)
 - open/close 3413 (time 1788815136.71 total 40.02 last 84.16)
 - open/close 4300 (time 1788815146.71 total 50.02 last 88.67)
 - open/close 5101 (time 1788815156.71 total 60.03 last 80.08)
 - open/close 5953 (time 1788815166.72 total 70.03 last 85.16)
 - open/close 6813 (time 1788815176.72 total 80.04 last 85.96)
 - open/close 7693 (time 1788815186.73 total 90.04 last 87.96)
 - open/close 8589 (time 1788815196.73 total 100.05 last 89.56)
 - open/close 9547 (time 1788815206.74 total 110.05 last 95.71)
total: 10000 open/close in 115.03 seconds: 86.93 ops/second
 - unlinked 0 (time 1788815214 ; total 0 ; last 0)
total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second
 - open/close 869 (time 1788815292.42 total 10.01 last 86.78)
 - open/close 1779 (time 1788815302.42 total 20.02 last 90.96)
 - open/close 2698 (time 1788815312.43 total 30.03 last 91.82)
 - open/close 3664 (time 1788815322.44 total 40.03 last 96.57)
 - open/close 4535 (time 1788815332.45 total 50.04 last 87.01)
 - open/close 5442 (time 1788815342.45 total 60.04 last 90.66)
 - open/close 6349 (time 1788815352.46 total 70.06 last 90.59)
 - open/close 7248 (time 1788815362.47 total 80.06 last 89.87)
 - open/close 8156 (time 1788815372.47 total 90.06 last 90.79)
 - open/close 9101 (time 1788815382.47 total 100.07 last 94.43)
total: 10000 open/close in 109.58 seconds: 91.26 ops/second
 - unlinked 0 (time 1788815394 ; total 0 ; last 0)
total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second
 - open/close 966 (time 1788815470.80 total 10.01 last 96.49)
 - open/close 1865 (time 1788815480.81 total 20.02 last 89.86)
 - open/close 2775 (time 1788815490.81 total 30.02 last 90.95)
 - open/close 3801 (time 1788815500.81 total 40.02 last 102.59)
 - open/close 4715 (time 1788815510.82 total 50.03 last 91.36)
 - open/close 5647 (time 1788815520.82 total 60.03 last 93.16)
 - open/close 6673 (time 1788815530.83 total 70.04 last 102.52)
 - open/close 7539 (time 1788815540.83 total 80.04 last 86.57)
 - open/close 8429 (time 1788815550.84 total 90.05 last 88.92)
 - open/close 9325 (time 1788815560.84 total 100.05 last 89.59)
total: 10000 open/close in 107.65 seconds: 92.90 ops/second
 - unlinked 0 (time 1788815571 ; total 0 ; last 0)
total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg412-server: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg412-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
On OST0, 5174 used inodes rc=0
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.112@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg412-server: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg412-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2