== conf-sanity test 69: replace an OST with the same index ========================================================== 12:22:19 (1788798139)
start mds service on oleg132-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  7 09:50 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: kmemleak disabled
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg132-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 636 (time 1788798242.80 total 10.01 last 63.54)
 - open/close 1279 (time 1788798252.80 total 20.01 last 64.30)
 - open/close 1919 (time 1788798262.81 total 30.02 last 63.95)
 - open/close 2427 (time 1788798272.82 total 40.03 last 50.73)
 - open/close 3043 (time 1788798282.85 total 50.06 last 61.44)
 - open/close 3708 (time 1788798292.87 total 60.08 last 66.36)
 - open/close 4359 (time 1788798302.89 total 70.09 last 65.01)
 - open/close 4985 (time 1788798312.91 total 80.11 last 62.48)
 - open/close 5654 (time 1788798322.93 total 90.13 last 66.75)
 - open/close 6365 (time 1788798332.94 total 100.15 last 71.00)
 - open/close 6991 (time 1788798342.95 total 110.15 last 62.57)
 - open/close 7647 (time 1788798352.96 total 120.16 last 65.54)
 - open/close 8214 (time 1788798362.97 total 130.18 last 56.60)
 - open/close 8814 (time 1788798372.97 total 140.18 last 60.00)
 - open/close 9505 (time 1788798382.99 total 150.19 last 69.00)
total: 10000 open/close in 157.24 seconds: 63.60 ops/second
 - unlinked 0 (time 1788798395 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 731 (time 1788798502.90 total 10.00 last 73.09)
 - open/close 1509 (time 1788798512.91 total 20.01 last 77.74)
 - open/close 2306 (time 1788798522.91 total 30.01 last 79.69)
 - open/close 3066 (time 1788798532.91 total 40.01 last 75.98)
 - open/close 3833 (time 1788798542.92 total 50.02 last 76.62)
 - open/close 4470 (time 1788798552.93 total 60.03 last 63.67)
 - open/close 5086 (time 1788798562.93 total 70.03 last 61.59)
 - open/close 5711 (time 1788798572.93 total 80.04 last 62.47)
 - open/close 6352 (time 1788798582.94 total 90.04 last 64.09)
 - open/close 7066 (time 1788798592.95 total 100.05 last 71.33)
 - open/close 7774 (time 1788798602.95 total 110.05 last 70.77)
 - open/close 8472 (time 1788798612.96 total 120.06 last 69.75)
 - open/close 8979 (time 1788798622.97 total 130.07 last 50.65)
 - open/close 9649 (time 1788798632.97 total 140.07 last 66.97)
total: 10000 open/close in 144.97 seconds: 68.98 ops/second
 - unlinked 0 (time 1788798642 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 669 (time 1788798751.22 total 10.01 last 66.83)
 - open/close 1346 (time 1788798761.25 total 20.04 last 67.51)
 - open/close 1997 (time 1788798771.26 total 30.05 last 65.03)
 - open/close 2681 (time 1788798781.26 total 40.05 last 68.40)
 - open/close 3340 (time 1788798791.26 total 50.05 last 65.88)
 - open/close 3964 (time 1788798801.27 total 60.06 last 62.34)
 - open/close 4519 (time 1788798811.28 total 70.07 last 55.44)
 - open/close 5252 (time 1788798821.29 total 80.08 last 73.25)
 - open/close 6048 (time 1788798831.30 total 90.09 last 79.50)
 - open/close 6741 (time 1788798841.30 total 100.09 last 69.29)
 - open/close 7563 (time 1788798851.30 total 110.10 last 82.19)
 - open/close 8390 (time 1788798861.31 total 120.10 last 82.69)
 - open/close 9220 (time 1788798871.31 total 130.10 last 82.94)
 - open/close 9892 (time 1788798881.33 total 140.12 last 67.10)
total: 10000 open/close in 142.19 seconds: 70.33 ops/second
 - unlinked 0 (time 1788798888 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 616 (time 1788799003.80 total 10.01 last 61.53)
 - open/close 1175 (time 1788799013.81 total 20.02 last 55.85)
 - open/close 1714 (time 1788799023.81 total 30.02 last 53.89)
 - open/close 2181 (time 1788799033.81 total 40.02 last 46.69)
 - open/close 2814 (time 1788799043.83 total 50.04 last 63.17)
 - open/close 3420 (time 1788799053.84 total 60.05 last 60.55)
 - open/close 3941 (time 1788799063.85 total 70.06 last 52.05)
 - open/close 4527 (time 1788799073.85 total 80.07 last 58.55)
 - open/close 5168 (time 1788799083.86 total 90.07 last 64.09)
 - open/close 5731 (time 1788799093.86 total 100.08 last 56.27)
 - open/close 6312 (time 1788799103.90 total 110.12 last 57.87)
 - open/close 6907 (time 1788799113.92 total 120.13 last 59.39)
 - open/close 7421 (time 1788799123.93 total 130.14 last 51.35)
 - open/close 8049 (time 1788799133.94 total 140.15 last 62.74)
 - open/close 8670 (time 1788799143.94 total 150.16 last 62.08)
 - open/close 9277 (time 1788799153.96 total 160.18 last 60.55)
 - open/close 9878 (time 1788799163.97 total 170.19 last 60.06)
total: 10000 open/close in 172.02 seconds: 58.13 ops/second
 - unlinked 0 (time 1788799170 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 494 (time 1788799286.96 total 10.01 last 49.33)
 - open/close 979 (time 1788799296.97 total 20.03 last 48.44)
 - open/close 1538 (time 1788799306.98 total 30.03 last 55.87)
 - open/close 2155 (time 1788799316.98 total 40.04 last 61.67)
 - open/close 2713 (time 1788799326.99 total 50.05 last 55.75)
 - open/close 3374 (time 1788799336.99 total 60.05 last 66.09)
 - open/close 3998 (time 1788799347.00 total 70.05 last 62.39)
 - open/close 4637 (time 1788799357.01 total 80.06 last 63.84)
 - open/close 5321 (time 1788799367.02 total 90.07 last 68.32)
 - open/close 5852 (time 1788799377.02 total 100.07 last 53.10)
 - open/close 6313 (time 1788799387.03 total 110.08 last 46.05)
 - open/close 6826 (time 1788799397.03 total 120.09 last 51.28)
 - open/close 7411 (time 1788799407.04 total 130.10 last 58.44)
 - open/close 7874 (time 1788799417.07 total 140.13 last 46.17)
 - open/close 8463 (time 1788799427.08 total 150.13 last 58.86)
 - open/close 9152 (time 1788799437.08 total 160.14 last 68.86)
 - open/close 9805 (time 1788799447.08 total 170.14 last 65.28)
total: 10000 open/close in 173.55 seconds: 57.62 ops/second
 - unlinked 0 (time 1788799455 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 416 (time 1788799562.05 total 10.00 last 41.59)
 - open/close 1054 (time 1788799572.06 total 20.01 last 63.75)
 - open/close 1702 (time 1788799582.07 total 30.02 last 64.74)
 - open/close 2338 (time 1788799592.08 total 40.03 last 63.54)
 - open/close 3008 (time 1788799602.09 total 50.04 last 66.91)
 - open/close 3705 (time 1788799612.10 total 60.05 last 69.67)
 - open/close 4335 (time 1788799622.11 total 70.06 last 62.93)
 - open/close 4901 (time 1788799632.11 total 80.06 last 56.58)
 - open/close 5489 (time 1788799642.13 total 90.08 last 58.72)
 - open/close 6060 (time 1788799652.13 total 100.08 last 57.10)
 - open/close 6673 (time 1788799662.13 total 110.08 last 61.30)
 - open/close 7224 (time 1788799672.13 total 120.08 last 55.06)
 - open/close 7834 (time 1788799682.16 total 130.11 last 60.82)
 - open/close 8459 (time 1788799692.17 total 140.12 last 62.47)
 - open/close 9171 (time 1788799702.17 total 150.12 last 71.16)
 - open/close 9893 (time 1788799712.18 total 160.12 last 72.19)
total: 10000 open/close in 161.71 seconds: 61.84 ops/second
 - unlinked 0 (time 1788799719 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 599 (time 1788799830.62 total 10.01 last 59.84)
 - open/close 1221 (time 1788799840.62 total 20.01 last 62.19)
 - open/close 1922 (time 1788799850.63 total 30.02 last 70.07)
 - open/close 2593 (time 1788799860.65 total 40.04 last 66.96)
 - open/close 3313 (time 1788799870.66 total 50.05 last 71.92)
 - open/close 3988 (time 1788799880.67 total 60.06 last 67.44)
 - open/close 4671 (time 1788799890.68 total 70.07 last 68.20)
 - open/close 5425 (time 1788799900.70 total 80.09 last 75.25)
 - open/close 6155 (time 1788799910.70 total 90.09 last 72.99)
 - open/close 6794 (time 1788799920.71 total 100.10 last 63.89)
 - open/close 7488 (time 1788799930.72 total 110.11 last 69.31)
 - open/close 8240 (time 1788799940.72 total 120.11 last 75.17)
 - open/close 8828 (time 1788799950.72 total 130.11 last 58.79)
 - open/close 9552 (time 1788799960.73 total 140.12 last 72.38)
total: 10000 open/close in 147.56 seconds: 67.77 ops/second
 - unlinked 0 (time 1788799972 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 620 (time 1788800078.85 total 10.01 last 61.95)
 - open/close 1211 (time 1788800088.85 total 20.01 last 59.10)
 - open/close 1918 (time 1788800098.85 total 30.01 last 70.68)
 - open/close 2509 (time 1788800108.87 total 40.02 last 59.02)
 - open/close 3053 (time 1788800118.87 total 50.03 last 54.37)
 - open/close 3665 (time 1788800128.87 total 60.03 last 61.18)
 - open/close 4293 (time 1788800138.88 total 70.04 last 62.77)
 - open/close 4931 (time 1788800148.88 total 80.04 last 63.80)
 - open/close 5645 (time 1788800158.89 total 90.04 last 71.35)
 - open/close 6344 (time 1788800168.89 total 100.05 last 69.86)
 - open/close 7062 (time 1788800178.89 total 110.05 last 71.80)
 - open/close 7782 (time 1788800188.89 total 120.05 last 71.99)
 - open/close 8463 (time 1788800198.92 total 130.08 last 67.89)
 - open/close 9134 (time 1788800208.94 total 140.10 last 67.00)
 - open/close 9898 (time 1788800218.95 total 150.10 last 76.34)
total: 10000 open/close in 151.75 seconds: 65.90 ops/second
 - unlinked 0 (time 1788800225 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 713 (time 1788800331.09 total 10.01 last 71.22)
 - open/close 1455 (time 1788800341.09 total 20.01 last 74.19)
 - open/close 2111 (time 1788800351.11 total 30.04 last 65.45)
 - open/close 2868 (time 1788800361.12 total 40.05 last 75.61)
 - open/close 3586 (time 1788800371.14 total 50.06 last 71.69)
 - open/close 4365 (time 1788800381.14 total 60.06 last 77.89)
 - open/close 5191 (time 1788800391.14 total 70.07 last 82.55)
 - open/close 5887 (time 1788800401.17 total 80.09 last 69.46)
 - open/close 6687 (time 1788800411.18 total 90.10 last 79.89)
 - open/close 7396 (time 1788800421.19 total 100.11 last 70.84)
 - open/close 8184 (time 1788800431.19 total 110.12 last 78.74)
 - open/close 9017 (time 1788800441.21 total 120.14 last 83.15)
 - open/close 9637 (time 1788800451.23 total 130.15 last 61.90)
total: 10000 open/close in 134.99 seconds: 74.08 ops/second
 - unlinked 0 (time 1788800459 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 706 (time 1788800568.95 total 10.00 last 70.57)
 - open/close 1416 (time 1788800578.95 total 20.01 last 70.98)
 - open/close 2156 (time 1788800588.95 total 30.01 last 73.99)
 - open/close 2881 (time 1788800598.96 total 40.02 last 72.44)
 - open/close 3581 (time 1788800608.97 total 50.02 last 69.96)
 - open/close 4118 (time 1788800618.98 total 60.04 last 53.62)
 - open/close 4775 (time 1788800628.98 total 70.04 last 65.69)
 - open/close 5370 (time 1788800639.00 total 80.05 last 59.40)
 - open/close 6131 (time 1788800649.00 total 90.05 last 76.10)
 - open/close 6891 (time 1788800659.01 total 100.07 last 75.90)
 - open/close 7629 (time 1788800669.02 total 110.08 last 73.74)
 - open/close 8405 (time 1788800679.03 total 120.09 last 77.52)
 - open/close 9117 (time 1788800689.04 total 130.09 last 71.15)
 - open/close 9848 (time 1788800699.04 total 140.10 last 73.08)
total: 10000 open/close in 142.92 seconds: 69.97 ops/second
 - unlinked 0 (time 1788800706 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
On OST0, 15318 used inodes rc=0
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.132@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.132@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.132@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server
stop mds service on oleg132-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server
unloading modules via unload_modules_local on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing unload_modules_local
oleg132-server: kmemleak disabled
modules unloaded.
pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2