== conf-sanity test 69: replace an OST with the same index ========================================================== 22:56:46 (1789181806)
start mds service on oleg220-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep 11 11:59 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg220-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 800 (time 1789181898.60 total 10.01 last 79.91)
 - open/close 1843 (time 1789181908.61 total 20.02 last 104.24)
 - open/close 2646 (time 1789181918.61 total 30.02 last 80.27)
 - open/close 3511 (time 1789181928.61 total 40.02 last 86.49)
 - open/close 4401 (time 1789181938.63 total 50.04 last 88.84)
 - open/close 5263 (time 1789181948.64 total 60.05 last 86.14)
 - open/close 5965 (time 1789181958.64 total 70.05 last 70.18)
 - open/close 6813 (time 1789181968.64 total 80.05 last 84.76)
 - open/close 7663 (time 1789181978.65 total 90.06 last 84.97)
 - open/close 8467 (time 1789181988.67 total 100.08 last 80.25)
 - open/close 9227 (time 1789181998.67 total 110.08 last 75.96)
 - open/close 9908 (time 1789182008.67 total 120.09 last 68.07)
total: 10000 open/close in 122.19 seconds: 81.84 ops/second
 - unlinked 0 (time 1789182016 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 804 (time 1789182105.43 total 10.01 last 80.33)
 - open/close 1594 (time 1789182115.43 total 20.01 last 79.00)
 - open/close 2460 (time 1789182125.44 total 30.01 last 86.58)
 - open/close 3285 (time 1789182135.44 total 40.01 last 82.48)
 - open/close 4133 (time 1789182145.44 total 50.02 last 84.76)
 - open/close 4816 (time 1789182155.45 total 60.02 last 68.27)
 - open/close 5627 (time 1789182165.46 total 70.03 last 81.02)
 - open/close 6483 (time 1789182175.47 total 80.04 last 85.49)
 - open/close 7254 (time 1789182185.48 total 90.05 last 77.04)
 - open/close 8212 (time 1789182195.48 total 100.06 last 95.77)
 - open/close 9057 (time 1789182205.48 total 110.06 last 84.46)
 - open/close 9869 (time 1789182215.49 total 120.06 last 81.19)
total: 10000 open/close in 121.32 seconds: 82.43 ops/second
 - unlinked 0 (time 1789182221 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 966 (time 1789182308.24 total 10.00 last 96.58)
 - open/close 1895 (time 1789182318.24 total 20.01 last 92.86)
 - open/close 2747 (time 1789182328.25 total 30.01 last 85.14)
 - open/close 3588 (time 1789182338.27 total 40.03 last 83.95)
 - open/close 4375 (time 1789182348.28 total 50.04 last 78.61)
 - open/close 5084 (time 1789182358.29 total 60.05 last 70.84)
 - open/close 5995 (time 1789182368.29 total 70.06 last 91.04)
 - open/close 6860 (time 1789182378.30 total 80.06 last 86.49)
 - open/close 7685 (time 1789182388.30 total 90.06 last 82.47)
 - open/close 8546 (time 1789182398.31 total 100.07 last 86.03)
 - open/close 9428 (time 1789182408.32 total 110.09 last 88.07)
total: 10000 open/close in 115.93 seconds: 86.26 ops/second
 - unlinked 0 (time 1789182418 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 814 (time 1789182510.84 total 10.01 last 81.33)
 - open/close 1710 (time 1789182520.84 total 20.01 last 89.59)
 - open/close 2572 (time 1789182530.84 total 30.01 last 86.16)
 - open/close 3482 (time 1789182540.85 total 40.03 last 90.90)
 - open/close 4223 (time 1789182550.86 total 50.03 last 74.07)
 - open/close 4995 (time 1789182560.87 total 60.04 last 77.13)
 - open/close 5810 (time 1789182570.87 total 70.05 last 81.43)
 - open/close 6507 (time 1789182580.88 total 80.06 last 69.63)
 - open/close 7122 (time 1789182590.90 total 90.07 last 61.40)
 - open/close 7810 (time 1789182600.91 total 100.08 last 68.77)
 - open/close 8701 (time 1789182610.91 total 110.08 last 89.05)
 - open/close 9522 (time 1789182620.92 total 120.09 last 82.05)
total: 10000 open/close in 124.71 seconds: 80.18 ops/second
 - unlinked 0 (time 1789182629 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 894 (time 1789182727.00 total 10.00 last 89.40)
 - open/close 1838 (time 1789182737.01 total 20.01 last 94.29)
 - open/close 2699 (time 1789182747.05 total 30.05 last 85.76)
 - open/close 3625 (time 1789182757.06 total 40.06 last 92.53)
 - open/close 4462 (time 1789182767.06 total 50.07 last 83.64)
 - open/close 5274 (time 1789182777.07 total 60.07 last 81.15)
 - open/close 6122 (time 1789182787.08 total 70.08 last 84.74)
 - open/close 6855 (time 1789182797.08 total 80.08 last 73.28)
 - open/close 7661 (time 1789182807.08 total 90.08 last 80.60)
 - open/close 8533 (time 1789182817.09 total 100.09 last 87.13)
 - open/close 9436 (time 1789182827.09 total 110.09 last 90.29)
total: 10000 open/close in 117.37 seconds: 85.20 ops/second
 - unlinked 0 (time 1789182839 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 837 (time 1789182934.80 total 10.01 last 83.63)
 - open/close 1734 (time 1789182944.80 total 20.02 last 89.63)
 - open/close 2547 (time 1789182954.81 total 30.02 last 81.24)
 - open/close 3375 (time 1789182964.82 total 40.03 last 82.75)
 - open/close 4189 (time 1789182974.82 total 50.04 last 81.36)
 - open/close 5007 (time 1789182984.82 total 60.04 last 81.79)
 - open/close 5788 (time 1789182994.83 total 70.04 last 78.05)
 - open/close 6604 (time 1789183004.84 total 80.06 last 81.49)
 - open/close 7348 (time 1789183014.85 total 90.07 last 74.33)
 - open/close 8100 (time 1789183024.86 total 100.07 last 75.17)
 - open/close 8825 (time 1789183034.86 total 110.08 last 72.46)
 - open/close 9605 (time 1789183044.89 total 120.10 last 77.80)
total: 10000 open/close in 125.61 seconds: 79.61 ops/second
 - unlinked 0 (time 1789183055 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 798 (time 1789183155.64 total 10.02 last 79.67)
 - open/close 1571 (time 1789183165.66 total 20.03 last 77.16)
 - open/close 2355 (time 1789183175.67 total 30.04 last 78.32)
 - open/close 3078 (time 1789183185.68 total 40.05 last 72.27)
 - open/close 3919 (time 1789183195.68 total 50.06 last 84.04)
 - open/close 4746 (time 1789183205.69 total 60.06 last 82.67)
 - open/close 5533 (time 1789183215.69 total 70.07 last 78.64)
 - open/close 6370 (time 1789183225.70 total 80.07 last 83.70)
 - open/close 7181 (time 1789183235.70 total 90.08 last 81.04)
 - open/close 7970 (time 1789183245.71 total 100.08 last 78.83)
 - open/close 8811 (time 1789183255.72 total 110.09 last 84.02)
 - open/close 9587 (time 1789183265.73 total 120.10 last 77.55)
total: 10000 open/close in 125.18 seconds: 79.89 ops/second
 - unlinked 0 (time 1789183275 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 751 (time 1789183374.43 total 10.02 last 74.98)
 - open/close 1350 (time 1789183384.46 total 20.04 last 59.73)
 - open/close 1917 (time 1789183394.46 total 30.04 last 56.70)
 - open/close 2626 (time 1789183404.46 total 40.05 last 70.88)
 - open/close 3541 (time 1789183414.46 total 50.05 last 91.48)
 - open/close 4369 (time 1789183424.48 total 60.07 last 82.64)
 - open/close 5259 (time 1789183434.48 total 70.07 last 88.97)
 - open/close 6071 (time 1789183444.49 total 80.07 last 81.17)
 - open/close 6771 (time 1789183454.49 total 90.07 last 69.99)
 - open/close 7631 (time 1789183464.50 total 100.08 last 85.93)
 - open/close 8464 (time 1789183474.50 total 110.09 last 83.28)
 - open/close 9139 (time 1789183484.50 total 120.09 last 67.48)
 - open/close 9754 (time 1789183494.52 total 130.11 last 61.40)
total: 10000 open/close in 134.20 seconds: 74.52 ops/second
 - unlinked 0 (time 1789183505 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 791 (time 1789183600.32 total 10.00 last 79.08)
 - open/close 1651 (time 1789183610.33 total 20.01 last 85.93)
 - open/close 2454 (time 1789183620.34 total 30.02 last 80.22)
 - open/close 3239 (time 1789183630.34 total 40.02 last 78.50)
 - open/close 4119 (time 1789183640.34 total 50.03 last 87.94)
 - open/close 4981 (time 1789183650.35 total 60.03 last 86.18)
 - open/close 5838 (time 1789183660.35 total 70.04 last 85.65)
 - open/close 6606 (time 1789183670.36 total 80.04 last 76.78)
 - open/close 7432 (time 1789183680.36 total 90.04 last 82.58)
 - open/close 8300 (time 1789183690.36 total 100.04 last 86.77)
 - open/close 9049 (time 1789183700.36 total 110.05 last 74.88)
 - open/close 9820 (time 1789183710.37 total 120.05 last 77.08)
total: 10000 open/close in 122.93 seconds: 81.35 ops/second
 - unlinked 0 (time 1789183718 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 803 (time 1789183811.31 total 10.00 last 80.29)
 - open/close 1553 (time 1789183821.32 total 20.00 last 74.98)
 - open/close 2288 (time 1789183831.33 total 30.02 last 73.41)
 - open/close 3027 (time 1789183841.33 total 40.02 last 73.87)
 - open/close 3740 (time 1789183851.34 total 50.03 last 71.22)
 - open/close 4392 (time 1789183861.36 total 60.04 last 65.12)
 - open/close 5132 (time 1789183871.38 total 70.06 last 73.84)
 - open/close 5841 (time 1789183881.38 total 80.07 last 70.89)
 - open/close 6681 (time 1789183891.38 total 90.07 last 83.98)
 - open/close 7475 (time 1789183901.39 total 100.07 last 79.34)
 - open/close 8267 (time 1789183911.40 total 110.09 last 79.11)
 - open/close 8991 (time 1789183921.41 total 120.09 last 72.34)
 - open/close 9963 (time 1789183931.41 total 130.10 last 97.15)
total: 10000 open/close in 130.38 seconds: 76.70 ops/second
 - unlinked 0 (time 1789183936 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg220-server: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg220-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg220-server: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg220-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
unloading modules via unload_modules_local on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing unload_modules_local
oleg220-server: kmemleak disabled
modules unloaded.
pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2