== conf-sanity test 69: replace an OST with the same index ========================================================== 22:20:20 (1790734820)
start mds service on oleg311-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1947032 Sep 29 13:13 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg311-server'
oleg311-server: oleg311-server.virtnet: executing load_modules_local
oleg311-server: kmemleak disabled
oleg311-server: Loading modules from /home/green/git/lustre-release/lustre
oleg311-server: detected 4 online CPUs by sysfs
oleg311-server: MODOPTS_LIBCFS=
oleg311-server: Force libcfs to create 2 CPU partitions
oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg311-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg311-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-OST0000
oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg311-client.virtnet:  -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 675 (time 1790734907.63 total 10.00 last 67.47)
 - open/close 1349 (time 1790734917.64 total 20.01 last 67.37)
 - open/close 1998 (time 1790734927.64 total 30.02 last 64.85)
 - open/close 2649 (time 1790734937.66 total 40.04 last 64.96)
 - open/close 3317 (time 1790734947.67 total 50.04 last 66.76)
 - open/close 4040 (time 1790734957.69 total 60.06 last 72.16)
 - open/close 4753 (time 1790734967.71 total 70.09 last 71.14)
 - open/close 5508 (time 1790734977.72 total 80.10 last 75.41)
 - open/close 6186 (time 1790734987.73 total 90.11 last 67.75)
 - open/close 6829 (time 1790734997.74 total 100.11 last 64.27)
 - open/close 7459 (time 1790735007.75 total 110.12 last 62.91)
 - open/close 8097 (time 1790735017.76 total 120.13 last 63.74)
 - open/close 8766 (time 1790735027.77 total 130.15 last 66.81)
 - open/close 9430 (time 1790735037.78 total 140.15 last 66.38)
total: 10000 open/close in 148.54 seconds: 67.32 ops/second
 - unlinked 0 (time 1790735050 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 643 (time 1790735162.64 total 10.00 last 64.28)
 - open/close 1229 (time 1790735172.65 total 20.01 last 58.57)
 - open/close 1893 (time 1790735182.66 total 30.02 last 66.34)
 - open/close 2552 (time 1790735192.66 total 40.02 last 65.85)
 - open/close 3199 (time 1790735202.67 total 50.03 last 64.63)
 - open/close 3890 (time 1790735212.68 total 60.04 last 69.06)
 - open/close 4595 (time 1790735222.69 total 70.05 last 70.46)
 - open/close 5216 (time 1790735232.70 total 80.06 last 62.01)
 - open/close 5757 (time 1790735242.72 total 90.08 last 53.99)
 - open/close 6357 (time 1790735252.73 total 100.09 last 59.96)
 - open/close 6978 (time 1790735262.73 total 110.09 last 62.09)
 - open/close 7683 (time 1790735272.74 total 120.10 last 70.45)
 - open/close 8305 (time 1790735282.74 total 130.10 last 62.16)
 - open/close 8919 (time 1790735292.75 total 140.11 last 61.37)
 - open/close 9568 (time 1790735302.76 total 150.12 last 64.82)
total: 10000 open/close in 157.15 seconds: 63.64 ops/second
 - unlinked 0 (time 1790735314 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 671 (time 1790735428.91 total 10.02 last 66.98)
 - open/close 1376 (time 1790735438.92 total 20.02 last 70.46)
 - open/close 2000 (time 1790735448.93 total 30.03 last 62.35)
 - open/close 2645 (time 1790735458.95 total 40.06 last 64.32)
 - open/close 3268 (time 1790735468.96 total 50.07 last 62.26)
 - open/close 3892 (time 1790735478.96 total 60.07 last 62.38)
 - open/close 4514 (time 1790735488.97 total 70.07 last 62.17)
 - open/close 5167 (time 1790735498.98 total 80.09 last 65.20)
 - open/close 5758 (time 1790735509.00 total 90.11 last 59.01)
 - open/close 6308 (time 1790735519.00 total 100.11 last 54.97)
 - open/close 6905 (time 1790735529.01 total 110.11 last 59.68)
 - open/close 7536 (time 1790735539.02 total 120.13 last 63.01)
 - open/close 8059 (time 1790735549.03 total 130.13 last 52.27)
 - open/close 8630 (time 1790735559.03 total 140.13 last 57.10)
 - open/close 9222 (time 1790735569.04 total 150.14 last 59.14)
 - open/close 9813 (time 1790735579.09 total 160.19 last 58.80)
total: 10000 open/close in 163.43 seconds: 61.19 ops/second
 - unlinked 0 (time 1790735586 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 605 (time 1790735695.70 total 10.01 last 60.44)
 - open/close 1212 (time 1790735705.70 total 20.01 last 60.69)
 - open/close 1834 (time 1790735715.72 total 30.04 last 62.05)
 - open/close 2483 (time 1790735725.73 total 40.04 last 64.86)
 - open/close 3091 (time 1790735735.75 total 50.06 last 60.68)
 - open/close 3720 (time 1790735745.75 total 60.06 last 62.89)
 - open/close 4356 (time 1790735755.75 total 70.06 last 63.59)
 - open/close 4992 (time 1790735765.76 total 80.08 last 63.53)
 - open/close 5637 (time 1790735775.78 total 90.09 last 64.38)
 - open/close 6256 (time 1790735785.79 total 100.10 last 61.88)
 - open/close 6931 (time 1790735795.81 total 110.12 last 67.33)
 - open/close 7623 (time 1790735805.82 total 120.13 last 69.13)
 - open/close 8288 (time 1790735815.83 total 130.14 last 66.44)
 - open/close 8916 (time 1790735825.84 total 140.16 last 62.71)
 - open/close 9598 (time 1790735835.85 total 150.16 last 68.14)
total: 10000 open/close in 156.05 seconds: 64.08 ops/second
 - unlinked 0 (time 1790735845 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 616 (time 1790735950.72 total 10.02 last 61.46)
 - open/close 1288 (time 1790735960.72 total 20.03 last 67.14)
 - open/close 2023 (time 1790735970.76 total 30.06 last 73.27)
 - open/close 2696 (time 1790735980.77 total 40.08 last 67.22)
 - open/close 3369 (time 1790735990.77 total 50.08 last 67.29)
 - open/close 4049 (time 1790736000.78 total 60.08 last 67.94)
 - open/close 4723 (time 1790736010.78 total 70.09 last 67.35)
 - open/close 5381 (time 1790736020.80 total 80.11 last 65.68)
 - open/close 6032 (time 1790736030.83 total 90.13 last 64.95)
 - open/close 6697 (time 1790736040.83 total 100.14 last 66.45)
 - open/close 7348 (time 1790736050.83 total 110.14 last 65.09)
 - open/close 8015 (time 1790736060.85 total 120.15 last 66.62)
 - open/close 8666 (time 1790736070.85 total 130.16 last 65.08)
 - open/close 9292 (time 1790736080.85 total 140.16 last 62.58)
 - open/close 9976 (time 1790736090.86 total 150.16 last 68.37)
total: 10000 open/close in 150.54 seconds: 66.43 ops/second
 - unlinked 0 (time 1790736095 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 680 (time 1790736203.27 total 10.00 last 67.99)
 - open/close 1301 (time 1790736213.27 total 20.01 last 62.06)
 - open/close 1930 (time 1790736223.28 total 30.01 last 62.86)
 - open/close 2618 (time 1790736233.29 total 40.03 last 68.72)
 - open/close 3283 (time 1790736243.31 total 50.05 last 66.36)
 - open/close 3976 (time 1790736253.31 total 60.05 last 69.30)
 - open/close 4615 (time 1790736263.33 total 70.06 last 63.82)
 - open/close 5198 (time 1790736273.33 total 80.06 last 58.29)
 - open/close 5834 (time 1790736283.35 total 90.08 last 63.47)
 - open/close 6463 (time 1790736293.35 total 100.09 last 62.88)
 - open/close 7073 (time 1790736303.36 total 110.10 last 60.94)
 - open/close 7685 (time 1790736313.38 total 120.12 last 61.08)
 - open/close 8355 (time 1790736323.39 total 130.12 last 66.97)
 - open/close 9040 (time 1790736333.40 total 140.13 last 68.42)
 - open/close 9705 (time 1790736343.41 total 150.14 last 66.44)
total: 10000 open/close in 155.18 seconds: 64.44 ops/second
 - unlinked 0 (time 1790736352 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 680 (time 1790736464.31 total 10.02 last 67.88)
 - open/close 1350 (time 1790736474.32 total 20.03 last 66.94)
 - open/close 2002 (time 1790736484.33 total 30.03 last 65.15)
 - open/close 2702 (time 1790736494.36 total 40.06 last 69.79)
 - open/close 3399 (time 1790736504.37 total 50.07 last 69.63)
 - open/close 4091 (time 1790736514.37 total 60.08 last 69.16)
 - open/close 4799 (time 1790736524.38 total 70.08 last 70.79)
 - open/close 5443 (time 1790736534.38 total 80.08 last 64.40)
 - open/close 6148 (time 1790736544.38 total 90.09 last 70.47)
 - open/close 6821 (time 1790736554.38 total 100.09 last 67.29)
 - open/close 7484 (time 1790736564.38 total 110.09 last 66.30)
 - open/close 8156 (time 1790736574.39 total 120.10 last 67.12)
 - open/close 8776 (time 1790736584.40 total 130.11 last 61.97)
 - open/close 9485 (time 1790736594.41 total 140.12 last 70.83)
total: 10000 open/close in 148.27 seconds: 67.44 ops/second
 - unlinked 0 (time 1790736606 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 636 (time 1790736717.28 total 10.00 last 63.60)
 - open/close 1319 (time 1790736727.29 total 20.01 last 68.26)
 - open/close 2021 (time 1790736737.29 total 30.01 last 70.17)
 - open/close 2714 (time 1790736747.29 total 40.01 last 69.30)
 - open/close 3313 (time 1790736757.32 total 50.04 last 59.74)
 - open/close 3867 (time 1790736767.32 total 60.04 last 55.38)
 - open/close 4433 (time 1790736777.34 total 70.06 last 56.48)
 - open/close 5077 (time 1790736787.38 total 80.10 last 64.19)
 - open/close 5761 (time 1790736797.39 total 90.11 last 68.32)
 - open/close 6451 (time 1790736807.40 total 100.12 last 68.94)
 - open/close 7091 (time 1790736817.41 total 110.13 last 63.94)
 - open/close 7734 (time 1790736827.41 total 120.13 last 64.25)
 - open/close 8394 (time 1790736837.42 total 130.14 last 65.95)
 - open/close 8985 (time 1790736847.43 total 140.15 last 59.03)
 - open/close 9549 (time 1790736857.44 total 150.16 last 56.35)
total: 10000 open/close in 157.73 seconds: 63.40 ops/second
 - unlinked 0 (time 1790736869 ; total 1 ; last 1)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 648 (time 1790736979.01 total 10.01 last 64.75)
 - open/close 1290 (time 1790736989.02 total 20.01 last 64.18)
 - open/close 1980 (time 1790736999.03 total 30.02 last 68.95)
 - open/close 2623 (time 1790737009.03 total 40.02 last 64.29)
 - open/close 3215 (time 1790737019.03 total 50.02 last 59.18)
 - open/close 3837 (time 1790737029.03 total 60.03 last 62.19)
 - open/close 4529 (time 1790737039.04 total 70.03 last 69.18)
 - open/close 5234 (time 1790737049.04 total 80.04 last 70.45)
 - open/close 5877 (time 1790737059.05 total 90.04 last 64.26)
 - open/close 6530 (time 1790737069.05 total 100.05 last 65.27)
 - open/close 7105 (time 1790737079.06 total 110.05 last 57.47)
 - open/close 7760 (time 1790737089.07 total 120.06 last 65.46)
 - open/close 8359 (time 1790737099.07 total 130.06 last 59.88)
 - open/close 8961 (time 1790737109.08 total 140.07 last 60.13)
 - open/close 9602 (time 1790737119.09 total 150.09 last 64.03)
total: 10000 open/close in 157.11 seconds: 63.65 ops/second
 - unlinked 0 (time 1790737131 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 625 (time 1790737245.61 total 10.00 last 62.49)
 - open/close 1284 (time 1790737255.61 total 20.00 last 65.89)
 - open/close 1865 (time 1790737265.65 total 30.04 last 57.87)
 - open/close 2479 (time 1790737275.67 total 40.06 last 61.28)
 - open/close 3135 (time 1790737285.67 total 50.07 last 65.58)
 - open/close 3799 (time 1790737295.68 total 60.08 last 66.35)
 - open/close 4490 (time 1790737305.68 total 70.08 last 69.07)
 - open/close 5232 (time 1790737315.71 total 80.11 last 74.00)
 - open/close 5813 (time 1790737325.71 total 90.11 last 58.08)
 - open/close 6404 (time 1790737335.73 total 100.12 last 59.03)
 - open/close 7065 (time 1790737345.73 total 110.13 last 66.08)
 - open/close 7686 (time 1790737355.74 total 120.13 last 62.04)
 - open/close 8306 (time 1790737365.74 total 130.14 last 61.97)
 - open/close 8988 (time 1790737375.75 total 140.14 last 68.18)
 - open/close 9639 (time 1790737385.76 total 150.16 last 65.01)
total: 10000 open/close in 155.77 seconds: 64.20 ops/second
 - unlinked 0 (time 1790737395 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg311-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg311-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg311-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-OST0000
oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg311-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg311-client.virtnet:  -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre
On OST0, 17862 used inodes rc=0
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg311-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg311-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.111@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.111@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.111@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg311-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-OST0000
oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg311-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg311-client.virtnet:  -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg311-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg311-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server
stop mds service on oleg311-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server
unloading modules via unload_modules_local on: 'oleg311-server'
oleg311-server: oleg311-server.virtnet: executing unload_modules_local
oleg311-server: kmemleak disabled
modules unloaded.
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2
pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2
pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2