== conf-sanity test 69: replace an OST with the same index ========================================================== 02:09:16 (1788847756)
start mds service on oleg448-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep  8 01:49 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg448-server'
oleg448-server: oleg448-server.virtnet: executing load_modules_local
oleg448-server: kmemleak disabled
oleg448-server: Loading modules from /home/green/git/lustre-release/lustre
oleg448-server: detected 4 online CPUs by sysfs
oleg448-server: MODOPTS_LIBCFS=
oleg448-server: Force libcfs to create 2 CPU partitions
oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg448-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg448-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-OST0000
oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg448-client.virtnet:  -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 567 (time 1788847843.81 total 10.02 last 56.59)
 - open/close 1180 (time 1788847853.82 total 20.03 last 61.22)
 - open/close 1825 (time 1788847863.82 total 30.04 last 64.49)
 - open/close 2558 (time 1788847873.83 total 40.04 last 73.27)
 - open/close 3209 (time 1788847883.83 total 50.04 last 65.08)
 - open/close 3899 (time 1788847893.83 total 60.05 last 68.99)
 - open/close 4446 (time 1788847903.84 total 70.06 last 54.63)
 - open/close 5144 (time 1788847913.85 total 80.06 last 69.77)
 - open/close 5847 (time 1788847923.86 total 90.07 last 70.21)
 - open/close 6511 (time 1788847933.86 total 100.07 last 66.40)
 - open/close 7130 (time 1788847943.87 total 110.08 last 61.84)
 - open/close 7832 (time 1788847953.87 total 120.09 last 70.19)
 - open/close 8432 (time 1788847963.88 total 130.09 last 59.96)
 - open/close 9009 (time 1788847973.88 total 140.10 last 57.67)
 - open/close 9627 (time 1788847983.90 total 150.11 last 61.73)
total: 10000 open/close in 156.69 seconds: 63.82 ops/second
 - unlinked 0 (time 1788847996 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 491 (time 1788848111.88 total 10.01 last 49.03)
 - open/close 1043 (time 1788848121.88 total 20.02 last 55.19)
 - open/close 1611 (time 1788848131.88 total 30.02 last 56.76)
 - open/close 2275 (time 1788848141.89 total 40.02 last 66.39)
 - open/close 2940 (time 1788848151.90 total 50.03 last 66.43)
 - open/close 3654 (time 1788848161.90 total 60.04 last 71.38)
 - open/close 4326 (time 1788848171.90 total 70.04 last 67.20)
 - open/close 4886 (time 1788848181.92 total 80.06 last 55.89)
 - open/close 5357 (time 1788848191.94 total 90.07 last 47.02)
 - open/close 5897 (time 1788848201.94 total 100.08 last 53.95)
 - open/close 6431 (time 1788848211.95 total 110.08 last 53.39)
 - open/close 7104 (time 1788848221.97 total 120.10 last 67.15)
 - open/close 7790 (time 1788848231.97 total 130.10 last 68.60)
 - open/close 8409 (time 1788848241.97 total 140.11 last 61.88)
 - open/close 8924 (time 1788848252.01 total 150.15 last 51.29)
 - open/close 9557 (time 1788848262.02 total 160.16 last 63.22)
total: 10000 open/close in 166.48 seconds: 60.07 ops/second
 - unlinked 0 (time 1788848272 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 659 (time 1788848382.79 total 10.01 last 65.85)
 - open/close 1283 (time 1788848392.80 total 20.01 last 62.35)
 - open/close 1884 (time 1788848402.83 total 30.04 last 59.94)
 - open/close 2605 (time 1788848412.84 total 40.05 last 72.04)
 - open/close 3367 (time 1788848422.85 total 50.06 last 76.10)
 - open/close 3971 (time 1788848432.87 total 60.08 last 60.28)
 - open/close 4598 (time 1788848442.87 total 70.08 last 62.69)
 - open/close 5196 (time 1788848452.89 total 80.10 last 59.69)
 - open/close 5795 (time 1788848462.89 total 90.11 last 59.87)
 - open/close 6396 (time 1788848472.91 total 100.12 last 60.03)
 - open/close 7003 (time 1788848482.91 total 110.13 last 60.66)
 - open/close 7743 (time 1788848492.92 total 120.13 last 73.97)
 - open/close 8374 (time 1788848502.93 total 130.14 last 63.02)
 - open/close 8939 (time 1788848512.94 total 140.15 last 56.44)
 - open/close 9477 (time 1788848522.96 total 150.17 last 53.69)
total: 10000 open/close in 159.17 seconds: 62.83 ops/second
 - unlinked 0 (time 1788848536 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 684 (time 1788848645.25 total 10.01 last 68.36)
 - open/close 1300 (time 1788848655.25 total 20.01 last 61.57)
 - open/close 1843 (time 1788848665.26 total 30.02 last 54.27)
 - open/close 2468 (time 1788848675.27 total 40.03 last 62.44)
 - open/close 3123 (time 1788848685.28 total 50.04 last 65.40)
 - open/close 3812 (time 1788848695.28 total 60.04 last 68.89)
 - open/close 4505 (time 1788848705.28 total 70.04 last 69.29)
 - open/close 5248 (time 1788848715.29 total 80.05 last 74.27)
 - open/close 5985 (time 1788848725.29 total 90.05 last 73.67)
 - open/close 6676 (time 1788848735.30 total 100.06 last 69.03)
 - open/close 7337 (time 1788848745.30 total 110.06 last 66.09)
 - open/close 8020 (time 1788848755.30 total 120.06 last 68.29)
 - open/close 8664 (time 1788848765.32 total 130.08 last 64.30)
 - open/close 9279 (time 1788848775.32 total 140.08 last 61.49)
total: 10000 open/close in 150.06 seconds: 66.64 ops/second
 - unlinked 0 (time 1788848789 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 726 (time 1788848900.07 total 10.01 last 72.53)
 - open/close 1414 (time 1788848910.09 total 20.02 last 68.71)
 - open/close 2087 (time 1788848920.09 total 30.02 last 67.30)
 - open/close 2755 (time 1788848930.09 total 40.03 last 66.76)
 - open/close 3456 (time 1788848940.10 total 50.03 last 70.09)
 - open/close 4148 (time 1788848950.12 total 60.05 last 69.05)
 - open/close 4848 (time 1788848960.12 total 70.06 last 69.96)
 - open/close 5554 (time 1788848970.13 total 80.07 last 70.55)
 - open/close 6232 (time 1788848980.14 total 90.07 last 67.74)
 - open/close 6881 (time 1788848990.15 total 100.08 last 64.84)
 - open/close 7575 (time 1788849000.15 total 110.08 last 69.39)
 - open/close 8245 (time 1788849010.16 total 120.10 last 66.92)
 - open/close 8855 (time 1788849020.19 total 130.13 last 60.80)
 - open/close 9558 (time 1788849030.20 total 140.13 last 70.28)
total: 10000 open/close in 146.33 seconds: 68.34 ops/second
 - unlinked 0 (time 1788849040 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 690 (time 1788849141.92 total 10.00 last 68.99)
 - open/close 1373 (time 1788849151.92 total 20.01 last 68.25)
 - open/close 2083 (time 1788849161.93 total 30.02 last 70.94)
 - open/close 2734 (time 1788849171.94 total 40.02 last 65.07)
 - open/close 3387 (time 1788849181.94 total 50.03 last 65.27)
 - open/close 4028 (time 1788849191.95 total 60.03 last 64.06)
 - open/close 4671 (time 1788849201.97 total 70.05 last 64.16)
 - open/close 5335 (time 1788849211.97 total 80.05 last 66.39)
 - open/close 5986 (time 1788849221.99 total 90.07 last 64.98)
 - open/close 6660 (time 1788849232.01 total 100.09 last 67.27)
 - open/close 7346 (time 1788849242.01 total 110.10 last 68.58)
 - open/close 8028 (time 1788849252.02 total 120.11 last 68.14)
 - open/close 8690 (time 1788849262.03 total 130.12 last 66.12)
 - open/close 9340 (time 1788849272.07 total 140.15 last 64.79)
 - open/close 9972 (time 1788849282.07 total 150.16 last 63.15)
total: 10000 open/close in 150.50 seconds: 66.44 ops/second
 - unlinked 0 (time 1788849286 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 738 (time 1788849390.20 total 10.01 last 73.74)
 - open/close 1444 (time 1788849400.21 total 20.02 last 70.52)
 - open/close 2170 (time 1788849410.21 total 30.02 last 72.59)
 - open/close 2865 (time 1788849420.22 total 40.03 last 69.44)
 - open/close 3550 (time 1788849430.24 total 50.04 last 68.41)
 - open/close 4125 (time 1788849440.24 total 60.05 last 57.47)
 - open/close 4810 (time 1788849450.25 total 70.05 last 68.45)
 - open/close 5498 (time 1788849460.26 total 80.07 last 68.72)
 - open/close 6172 (time 1788849470.26 total 90.07 last 67.38)
 - open/close 6876 (time 1788849480.27 total 100.08 last 70.34)
 - open/close 7547 (time 1788849490.28 total 110.09 last 67.03)
 - open/close 8266 (time 1788849500.29 total 120.10 last 71.83)
 - open/close 8983 (time 1788849510.31 total 130.11 last 71.61)
 - open/close 9730 (time 1788849520.31 total 140.11 last 74.68)
total: 10000 open/close in 143.51 seconds: 69.68 ops/second
 - unlinked 0 (time 1788849526 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 745 (time 1788849617.48 total 10.01 last 74.42)
 - open/close 1501 (time 1788849627.49 total 20.02 last 75.52)
 - open/close 2271 (time 1788849637.49 total 30.02 last 76.98)
 - open/close 2988 (time 1788849647.49 total 40.03 last 71.68)
 - open/close 3688 (time 1788849657.50 total 50.03 last 69.98)
 - open/close 4414 (time 1788849667.51 total 60.04 last 72.52)
 - open/close 5108 (time 1788849677.53 total 70.06 last 69.24)
 - open/close 5819 (time 1788849687.54 total 80.07 last 71.06)
 - open/close 6566 (time 1788849697.55 total 90.09 last 74.59)
 - open/close 7278 (time 1788849707.56 total 100.09 last 71.15)
 - open/close 8027 (time 1788849717.57 total 110.10 last 74.84)
 - open/close 8780 (time 1788849727.57 total 120.10 last 75.30)
 - open/close 9578 (time 1788849737.57 total 130.11 last 79.76)
total: 10000 open/close in 135.11 seconds: 74.02 ops/second
 - unlinked 0 (time 1788849745 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 850 (time 1788849833.14 total 10.01 last 84.92)
 - open/close 1647 (time 1788849843.15 total 20.02 last 79.64)
 - open/close 2404 (time 1788849853.15 total 30.02 last 75.66)
 - open/close 3151 (time 1788849863.17 total 40.04 last 74.55)
 - open/close 3828 (time 1788849873.18 total 50.05 last 67.66)
 - open/close 4540 (time 1788849883.18 total 60.05 last 71.15)
 - open/close 5289 (time 1788849893.19 total 70.06 last 74.84)
 - open/close 6031 (time 1788849903.19 total 80.07 last 74.18)
 - open/close 6793 (time 1788849913.20 total 90.07 last 76.15)
 - open/close 7500 (time 1788849923.20 total 100.07 last 70.68)
 - open/close 8250 (time 1788849933.21 total 110.08 last 74.97)
 - open/close 8994 (time 1788849943.21 total 120.08 last 74.40)
 - open/close 9834 (time 1788849953.22 total 130.09 last 83.91)
total: 10000 open/close in 132.10 seconds: 75.70 ops/second
 - unlinked 0 (time 1788849958 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 776 (time 1788850042.05 total 10.01 last 77.53)
 - open/close 1560 (time 1788850052.06 total 20.02 last 78.32)
 - open/close 2313 (time 1788850062.06 total 30.02 last 75.28)
 - open/close 3059 (time 1788850072.07 total 40.03 last 74.56)
 - open/close 3873 (time 1788850082.07 total 50.03 last 81.39)
 - open/close 4730 (time 1788850092.07 total 60.03 last 85.69)
 - open/close 5616 (time 1788850102.08 total 70.03 last 88.55)
 - open/close 6526 (time 1788850112.08 total 80.04 last 90.97)
 - open/close 7499 (time 1788850122.09 total 90.05 last 97.21)
 - open/close 8377 (time 1788850132.09 total 100.05 last 87.75)
 - open/close 9230 (time 1788850142.10 total 110.06 last 85.27)
total: 10000 open/close in 118.70 seconds: 84.24 ops/second
 - unlinked 0 (time 1788850153 ; total 0 ; last 0)
total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg448-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg448-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.148@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg448-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-OST0000
oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg448-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg448-client.virtnet:  -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre
On OST0, 19414 used inodes rc=0
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg448-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg448-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.148@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.148@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.148@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.204.148@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg448-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-OST0000
oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg448-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg448-client.virtnet:  -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg448-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg448-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server
stop mds service on oleg448-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server
unloading modules via unload_modules_local on: 'oleg448-server'
oleg448-server: oleg448-server.virtnet: executing unload_modules_local
oleg448-server: kmemleak disabled
modules unloaded.
pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2
pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2