== conf-sanity test 69: replace an OST with the same index ========================================================== 20:41:12 (1789778472)
start mds service on oleg110-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing load_modules_local
oleg110-server: kmemleak disabled
oleg110-server: Loading modules from /home/green/git/lustre-release/lustre
oleg110-server: detected 4 online CPUs by sysfs
oleg110-server: MODOPTS_LIBCFS=
oleg110-server: Force libcfs to create 2 CPU partitions
oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 856 (time 1789778553.28 total 10.00 last 85.59)
 - open/close 1580 (time 1789778563.29 total 20.01 last 72.35)
 - open/close 2345 (time 1789778573.30 total 30.02 last 76.41)
 - open/close 3093 (time 1789778583.30 total 40.02 last 74.77)
 - open/close 3905 (time 1789778593.31 total 50.03 last 81.13)
 - open/close 4650 (time 1789778603.32 total 60.04 last 74.42)
 - open/close 5365 (time 1789778613.33 total 70.05 last 71.44)
 - open/close 6184 (time 1789778623.34 total 80.06 last 81.78)
 - open/close 6833 (time 1789778633.35 total 90.07 last 64.87)
 - open/close 7357 (time 1789778643.35 total 100.07 last 52.38)
 - open/close 8013 (time 1789778653.37 total 110.09 last 65.49)
 - open/close 8768 (time 1789778663.39 total 120.11 last 75.33)
 - open/close 9666 (time 1789778673.40 total 130.13 last 89.68)
total: 10000 open/close in 134.42 seconds: 74.40 ops/second
 - unlinked 0 (time 1789778681 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 709 (time 1789778776.00 total 10.01 last 70.84)
 - open/close 1440 (time 1789778786.00 total 20.01 last 73.08)
 - open/close 2159 (time 1789778796.01 total 30.02 last 71.86)
 - open/close 3019 (time 1789778806.03 total 40.04 last 85.76)
 - open/close 3894 (time 1789778816.04 total 50.05 last 87.43)
 - open/close 4715 (time 1789778826.05 total 60.06 last 82.02)
 - open/close 5640 (time 1789778836.05 total 70.06 last 92.48)
 - open/close 6396 (time 1789778846.06 total 80.07 last 75.58)
 - open/close 7146 (time 1789778856.06 total 90.07 last 74.97)
 - open/close 7959 (time 1789778866.07 total 100.08 last 81.22)
 - open/close 8716 (time 1789778876.07 total 110.08 last 75.67)
 - open/close 9435 (time 1789778886.10 total 120.11 last 71.72)
total: 10000 open/close in 128.07 seconds: 78.08 ops/second
 - unlinked 0 (time 1789778899 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 871 (time 1789778995.69 total 10.01 last 87.02)
 - open/close 1559 (time 1789779005.69 total 20.01 last 68.77)
 - open/close 2261 (time 1789779015.70 total 30.02 last 70.18)
 - open/close 2952 (time 1789779025.70 total 40.02 last 69.07)
 - open/close 3733 (time 1789779035.71 total 50.03 last 78.06)
 - open/close 4522 (time 1789779045.71 total 60.03 last 78.89)
 - open/close 5289 (time 1789779055.72 total 70.04 last 76.61)
 - open/close 6004 (time 1789779065.72 total 80.04 last 71.47)
 - open/close 6710 (time 1789779075.72 total 90.05 last 70.58)
 - open/close 7342 (time 1789779085.73 total 100.05 last 63.16)
 - open/close 8081 (time 1789779095.74 total 110.06 last 73.86)
 - open/close 8803 (time 1789779105.75 total 120.07 last 72.08)
 - open/close 9565 (time 1789779115.76 total 130.09 last 76.11)
total: 10000 open/close in 135.54 seconds: 73.78 ops/second
 - unlinked 0 (time 1789779125 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 783 (time 1789779228.59 total 10.01 last 78.24)
 - open/close 1588 (time 1789779238.59 total 20.01 last 80.49)
 - open/close 2379 (time 1789779248.61 total 30.03 last 78.95)
 - open/close 3119 (time 1789779258.62 total 40.04 last 73.94)
 - open/close 3874 (time 1789779268.63 total 50.04 last 75.46)
 - open/close 4424 (time 1789779278.64 total 60.06 last 54.91)
 - open/close 5107 (time 1789779288.64 total 70.06 last 68.30)
 - open/close 5896 (time 1789779298.65 total 80.06 last 78.88)
 - open/close 6662 (time 1789779308.65 total 90.07 last 76.57)
 - open/close 7444 (time 1789779318.65 total 100.07 last 78.18)
 - open/close 8172 (time 1789779328.66 total 110.07 last 72.76)
 - open/close 8934 (time 1789779338.66 total 120.07 last 76.20)
 - open/close 9606 (time 1789779348.66 total 130.07 last 67.20)
total: 10000 open/close in 135.12 seconds: 74.01 ops/second
 - unlinked 0 (time 1789779358 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 858 (time 1789779459.25 total 10.01 last 85.72)
 - open/close 1675 (time 1789779469.25 total 20.02 last 81.64)
 - open/close 2377 (time 1789779479.25 total 30.02 last 70.19)
 - open/close 3047 (time 1789779489.26 total 40.02 last 66.98)
 - open/close 3759 (time 1789779499.26 total 50.03 last 71.14)
 - open/close 4603 (time 1789779509.27 total 60.04 last 84.34)
 - open/close 5413 (time 1789779519.28 total 70.04 last 80.95)
 - open/close 6166 (time 1789779529.28 total 80.04 last 75.28)
 - open/close 6956 (time 1789779539.29 total 90.05 last 78.92)
 - open/close 7679 (time 1789779549.29 total 100.06 last 72.28)
 - open/close 8443 (time 1789779559.29 total 110.06 last 76.40)
 - open/close 9194 (time 1789779569.31 total 120.07 last 75.02)
 - open/close 9974 (time 1789779579.32 total 130.09 last 77.88)
total: 10000 open/close in 130.35 seconds: 76.72 ops/second
 - unlinked 0 (time 1789779584 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 852 (time 1789779684.15 total 10.01 last 85.09)
 - open/close 1742 (time 1789779694.15 total 20.02 last 88.97)
 - open/close 2559 (time 1789779704.16 total 30.02 last 81.66)
 - open/close 3423 (time 1789779714.16 total 40.03 last 86.36)
 - open/close 4200 (time 1789779724.16 total 50.03 last 77.69)
 - open/close 5141 (time 1789779734.17 total 60.03 last 94.09)
 - open/close 5990 (time 1789779744.17 total 70.04 last 84.83)
 - open/close 6797 (time 1789779754.18 total 80.04 last 80.69)
 - open/close 7623 (time 1789779764.18 total 90.04 last 82.58)
 - open/close 8370 (time 1789779774.18 total 100.04 last 74.67)
 - open/close 9155 (time 1789779784.18 total 110.05 last 78.48)
total: 10000 open/close in 119.24 seconds: 83.86 ops/second
 - unlinked 0 (time 1789779797 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 877 (time 1789779889.02 total 10.00 last 87.67)
 - open/close 1725 (time 1789779899.02 total 20.01 last 84.78)
 - open/close 2667 (time 1789779909.03 total 30.01 last 94.12)
 - open/close 3505 (time 1789779919.03 total 40.02 last 83.76)
 - open/close 4288 (time 1789779929.03 total 50.02 last 78.30)
 - open/close 5106 (time 1789779939.05 total 60.04 last 81.66)
 - open/close 5831 (time 1789779949.06 total 70.04 last 72.47)
 - open/close 6733 (time 1789779959.06 total 80.05 last 90.14)
 - open/close 7509 (time 1789779969.07 total 90.06 last 77.52)
 - open/close 8291 (time 1789779979.07 total 100.06 last 78.19)
 - open/close 8987 (time 1789779989.08 total 110.07 last 69.53)
 - open/close 9837 (time 1789779999.10 total 120.09 last 84.84)
total: 10000 open/close in 122.76 seconds: 81.46 ops/second
 - unlinked 0 (time 1789780006 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 827 (time 1789780099.64 total 10.01 last 82.64)
 - open/close 1746 (time 1789780109.65 total 20.01 last 91.84)
 - open/close 2678 (time 1789780119.65 total 30.02 last 93.15)
 - open/close 3509 (time 1789780129.66 total 40.03 last 82.99)
 - open/close 4388 (time 1789780139.67 total 50.03 last 87.89)
 - open/close 5233 (time 1789780149.68 total 60.05 last 84.40)
 - open/close 6062 (time 1789780159.68 total 70.05 last 82.86)
 - open/close 6835 (time 1789780169.69 total 80.06 last 77.25)
 - open/close 7694 (time 1789780179.69 total 90.06 last 85.89)
 - open/close 8553 (time 1789780189.69 total 100.06 last 85.89)
 - open/close 9315 (time 1789780199.70 total 110.07 last 76.13)
total: 10000 open/close in 117.69 seconds: 84.97 ops/second
 - unlinked 0 (time 1789780211 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 868 (time 1789780300.53 total 10.01 last 86.74)
 - open/close 1731 (time 1789780310.54 total 20.02 last 86.19)
 - open/close 2584 (time 1789780320.54 total 30.02 last 85.28)
 - open/close 3280 (time 1789780330.55 total 40.03 last 69.56)
 - open/close 4076 (time 1789780340.55 total 50.03 last 79.58)
 - open/close 4804 (time 1789780350.55 total 60.03 last 72.80)
 - open/close 5588 (time 1789780360.55 total 70.03 last 78.38)
 - open/close 6445 (time 1789780370.56 total 80.04 last 85.60)
 - open/close 7262 (time 1789780380.57 total 90.05 last 81.65)
 - open/close 7983 (time 1789780390.57 total 100.05 last 72.09)
 - open/close 8822 (time 1789780400.58 total 110.06 last 83.86)
 - open/close 9626 (time 1789780410.58 total 120.07 last 80.33)
total: 10000 open/close in 124.57 seconds: 80.28 ops/second
 - unlinked 0 (time 1789780419 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 770 (time 1789780505.71 total 10.01 last 76.95)
 - open/close 1648 (time 1789780515.72 total 20.01 last 87.76)
 - open/close 2463 (time 1789780525.73 total 30.02 last 81.42)
 - open/close 3279 (time 1789780535.73 total 40.02 last 81.56)
 - open/close 4081 (time 1789780545.73 total 50.03 last 80.18)
 - open/close 4886 (time 1789780555.73 total 60.03 last 80.50)
 - open/close 5672 (time 1789780565.74 total 70.03 last 78.54)
 - open/close 6535 (time 1789780575.75 total 80.04 last 86.23)
 - open/close 7441 (time 1789780585.75 total 90.05 last 90.55)
 - open/close 8294 (time 1789780595.76 total 100.05 last 85.27)
 - open/close 9152 (time 1789780605.77 total 110.06 last 85.72)
 - open/close 9993 (time 1789780615.78 total 120.07 last 84.03)
total: 10000 open/close in 120.15 seconds: 83.23 ops/second
 - unlinked 0 (time 1789780620 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg110-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.110@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.110@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.110@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg110-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
unloading modules via unload_modules_local on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing unload_modules_local
oleg110-server: kmemleak disabled
modules unloaded.
pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2