== conf-sanity test 69: replace an OST with the same index ========================================================== 21:35:54 (1789781754)
start mds service on oleg329-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg329-server'
oleg329-server: oleg329-server.virtnet: executing load_modules_local
oleg329-server: kmemleak disabled
oleg329-server: Loading modules from /home/green/git/lustre-release/lustre
oleg329-server: detected 4 online CPUs by sysfs
oleg329-server: MODOPTS_LIBCFS=
oleg329-server: Force libcfs to create 2 CPU partitions
oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg329-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg329-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1
Started lustre-OST0000
oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg329-client.virtnet:  -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 801 (time 1789781835.48 total 10.00 last 80.09)
 - open/close 1622 (time 1789781845.48 total 20.00 last 82.09)
 - open/close 2388 (time 1789781855.50 total 30.02 last 76.48)
 - open/close 3229 (time 1789781865.50 total 40.02 last 84.06)
 - open/close 4076 (time 1789781875.51 total 50.03 last 84.66)
 - open/close 4933 (time 1789781885.51 total 60.03 last 85.67)
 - open/close 5791 (time 1789781895.51 total 70.03 last 85.79)
 - open/close 6591 (time 1789781905.53 total 80.04 last 79.92)
 - open/close 7318 (time 1789781915.53 total 90.05 last 72.65)
 - open/close 8178 (time 1789781925.54 total 100.06 last 85.90)
 - open/close 8890 (time 1789781935.56 total 110.08 last 71.11)
 - open/close 9630 (time 1789781945.56 total 120.08 last 74.00)
total: 10000 open/close in 124.73 seconds: 80.17 ops/second
 - unlinked 0 (time 1789781954 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 806 (time 1789782047.75 total 10.01 last 80.53)
 - open/close 1618 (time 1789782057.75 total 20.01 last 81.17)
 - open/close 2279 (time 1789782067.76 total 30.02 last 66.06)
 - open/close 3000 (time 1789782077.76 total 40.02 last 72.06)
 - open/close 3763 (time 1789782087.77 total 50.03 last 76.27)
 - open/close 4610 (time 1789782097.77 total 60.03 last 84.66)
 - open/close 5397 (time 1789782107.78 total 70.04 last 78.60)
 - open/close 6152 (time 1789782117.79 total 80.05 last 75.48)
 - open/close 6936 (time 1789782127.79 total 90.05 last 78.35)
 - open/close 7777 (time 1789782137.79 total 100.06 last 84.08)
 - open/close 8566 (time 1789782147.81 total 110.07 last 78.78)
 - open/close 9253 (time 1789782157.82 total 120.08 last 68.63)
 - open/close 9949 (time 1789782167.83 total 130.09 last 69.56)
total: 10000 open/close in 130.86 seconds: 76.42 ops/second
 - unlinked 0 (time 1789782172 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 775 (time 1789782276.63 total 10.00 last 77.50)
 - open/close 1523 (time 1789782286.64 total 20.01 last 74.76)
 - open/close 2198 (time 1789782296.64 total 30.01 last 67.50)
 - open/close 2911 (time 1789782306.64 total 40.01 last 71.28)
 - open/close 3620 (time 1789782316.65 total 50.01 last 70.86)
 - open/close 4255 (time 1789782326.66 total 60.03 last 63.39)
 - open/close 4896 (time 1789782336.67 total 70.04 last 64.07)
 - open/close 5592 (time 1789782346.67 total 80.04 last 69.58)
 - open/close 6334 (time 1789782356.68 total 90.04 last 74.16)
 - open/close 6970 (time 1789782366.69 total 100.06 last 63.52)
 - open/close 7606 (time 1789782376.69 total 110.06 last 63.57)
 - open/close 8355 (time 1789782386.70 total 120.07 last 74.85)
 - open/close 9064 (time 1789782396.70 total 130.07 last 70.86)
 - open/close 9808 (time 1789782406.71 total 140.08 last 74.35)
total: 10000 open/close in 142.09 seconds: 70.38 ops/second
 - unlinked 0 (time 1789782413 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 576 (time 1789782520.87 total 10.00 last 57.58)
 - open/close 1347 (time 1789782530.88 total 20.01 last 77.06)
 - open/close 1932 (time 1789782540.89 total 30.02 last 58.43)
 - open/close 2502 (time 1789782550.90 total 40.04 last 56.91)
 - open/close 3191 (time 1789782560.91 total 50.04 last 68.86)
 - open/close 3890 (time 1789782570.92 total 60.05 last 69.86)
 - open/close 4531 (time 1789782580.92 total 70.05 last 64.07)
 - open/close 5177 (time 1789782590.92 total 80.05 last 64.59)
 - open/close 5911 (time 1789782600.93 total 90.06 last 73.37)
 - open/close 6640 (time 1789782610.93 total 100.06 last 72.90)
 - open/close 7285 (time 1789782620.93 total 110.06 last 64.49)
 - open/close 7956 (time 1789782630.93 total 120.06 last 67.06)
 - open/close 8631 (time 1789782640.94 total 130.07 last 67.49)
 - open/close 9379 (time 1789782650.94 total 140.07 last 74.77)
total: 10000 open/close in 148.49 seconds: 67.35 ops/second
 - unlinked 0 (time 1789782664 ; total 1 ; last 1)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 716 (time 1789782769.67 total 10.01 last 71.51)
 - open/close 1443 (time 1789782779.67 total 20.01 last 72.70)
 - open/close 2159 (time 1789782789.67 total 30.02 last 71.55)
 - open/close 2929 (time 1789782799.69 total 40.04 last 76.88)
 - open/close 3597 (time 1789782809.70 total 50.05 last 66.74)
 - open/close 4271 (time 1789782819.71 total 60.06 last 67.32)
 - open/close 5030 (time 1789782829.72 total 70.07 last 75.79)
 - open/close 5834 (time 1789782839.75 total 80.10 last 80.20)
 - open/close 6547 (time 1789782849.76 total 90.11 last 71.20)
 - open/close 7230 (time 1789782859.77 total 100.12 last 68.25)
 - open/close 7966 (time 1789782869.77 total 110.12 last 73.57)
 - open/close 8676 (time 1789782879.79 total 120.14 last 70.89)
 - open/close 9402 (time 1789782889.79 total 130.14 last 72.57)
total: 10000 open/close in 139.88 seconds: 71.49 ops/second
 - unlinked 0 (time 1789782904 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 753 (time 1789782996.48 total 10.00 last 75.28)
 - open/close 1458 (time 1789783006.48 total 20.01 last 70.46)
 - open/close 2342 (time 1789783016.49 total 30.01 last 88.35)
 - open/close 3206 (time 1789783026.49 total 40.02 last 86.37)
 - open/close 4096 (time 1789783036.49 total 50.02 last 88.98)
 - open/close 4982 (time 1789783046.50 total 60.03 last 88.51)
 - open/close 5757 (time 1789783056.51 total 70.04 last 77.43)
 - open/close 6583 (time 1789783066.51 total 80.04 last 82.59)
 - open/close 7320 (time 1789783076.52 total 90.04 last 73.67)
 - open/close 8074 (time 1789783086.52 total 100.05 last 75.38)
 - open/close 8834 (time 1789783096.53 total 110.06 last 75.94)
 - open/close 9442 (time 1789783106.55 total 120.08 last 60.66)
total: 10000 open/close in 128.70 seconds: 77.70 ops/second
 - unlinked 0 (time 1789783120 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 772 (time 1789783215.25 total 10.01 last 77.11)
 - open/close 1572 (time 1789783225.27 total 20.03 last 79.89)
 - open/close 2410 (time 1789783235.27 total 30.03 last 83.78)
 - open/close 3228 (time 1789783245.29 total 40.04 last 81.67)
 - open/close 3992 (time 1789783255.29 total 50.04 last 76.39)
 - open/close 4669 (time 1789783265.29 total 60.05 last 67.66)
 - open/close 5573 (time 1789783275.30 total 70.06 last 90.34)
 - open/close 6355 (time 1789783285.30 total 80.06 last 78.16)
 - open/close 7047 (time 1789783295.31 total 90.07 last 69.18)
 - open/close 7718 (time 1789783305.31 total 100.07 last 67.08)
 - open/close 8528 (time 1789783315.33 total 110.08 last 80.88)
 - open/close 9337 (time 1789783325.33 total 120.09 last 80.83)
total: 10000 open/close in 128.90 seconds: 77.58 ops/second
 - unlinked 0 (time 1789783338 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 791 (time 1789783434.42 total 10.03 last 78.83)
 - open/close 1495 (time 1789783444.42 total 20.03 last 70.40)
 - open/close 2224 (time 1789783454.43 total 30.04 last 72.83)
 - open/close 2931 (time 1789783464.44 total 40.06 last 70.62)
 - open/close 3597 (time 1789783474.45 total 50.07 last 66.54)
 - open/close 4390 (time 1789783484.47 total 60.08 last 79.17)
 - open/close 5131 (time 1789783494.47 total 70.08 last 74.10)
 - open/close 5754 (time 1789783504.48 total 80.09 last 62.26)
 - open/close 6309 (time 1789783514.48 total 90.09 last 55.47)
 - open/close 7037 (time 1789783524.49 total 100.10 last 72.75)
 - open/close 7884 (time 1789783534.49 total 110.10 last 84.67)
 - open/close 8640 (time 1789783544.49 total 120.10 last 75.59)
 - open/close 9407 (time 1789783554.50 total 130.11 last 76.65)
total: 10000 open/close in 138.57 seconds: 72.17 ops/second
 - unlinked 0 (time 1789783567 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 797 (time 1789783662.92 total 10.01 last 79.61)
 - open/close 1573 (time 1789783672.93 total 20.01 last 77.58)
 - open/close 2390 (time 1789783682.93 total 30.02 last 81.68)
 - open/close 3205 (time 1789783692.93 total 40.02 last 81.46)
 - open/close 3958 (time 1789783702.95 total 50.04 last 75.17)
 - open/close 4634 (time 1789783712.97 total 60.05 last 67.50)
 - open/close 5094 (time 1789783722.97 total 70.06 last 45.98)
 - open/close 5718 (time 1789783732.98 total 80.06 last 62.36)
 - open/close 6437 (time 1789783742.99 total 90.08 last 71.82)
 - open/close 7195 (time 1789783753.00 total 100.09 last 75.68)
 - open/close 7873 (time 1789783763.01 total 110.10 last 67.73)
 - open/close 8637 (time 1789783773.01 total 120.10 last 76.39)
 - open/close 9404 (time 1789783783.02 total 130.11 last 76.62)
total: 10000 open/close in 137.13 seconds: 72.92 ops/second
 - unlinked 0 (time 1789783793 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 779 (time 1789783891.22 total 10.02 last 77.73)
 - open/close 1503 (time 1789783901.22 total 20.02 last 72.39)
 - open/close 2320 (time 1789783911.24 total 30.04 last 81.54)
 - open/close 3090 (time 1789783921.25 total 40.05 last 76.93)
 - open/close 3767 (time 1789783931.31 total 50.12 last 67.24)
 - open/close 4650 (time 1789783941.33 total 60.13 last 88.21)
 - open/close 5486 (time 1789783951.34 total 70.15 last 83.46)
 - open/close 6255 (time 1789783961.35 total 80.15 last 76.86)
 - open/close 7036 (time 1789783971.36 total 90.16 last 78.04)
 - open/close 7824 (time 1789783981.36 total 100.16 last 78.79)
 - open/close 8553 (time 1789783991.36 total 110.17 last 72.86)
 - open/close 9329 (time 1789784001.37 total 120.17 last 77.56)
total: 10000 open/close in 128.13 seconds: 78.05 ops/second
 - unlinked 0 (time 1789784013 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg329-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg329-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.129@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.129@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg329-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1
Started lustre-OST0000
oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg329-server: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg329-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg329-client.virtnet:  -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg329-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg329-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.129@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.129@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.129@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.129@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.129@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg329-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1
Started lustre-OST0000
oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg329-server: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg329-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg329-client.virtnet:  -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg329-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg329-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server
stop mds service on oleg329-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server
unloading modules via unload_modules_local on: 'oleg329-server'
oleg329-server: oleg329-server.virtnet: executing unload_modules_local
oleg329-server: kmemleak disabled
modules unloaded.
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2
pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2
pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2
pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2