== conf-sanity test 69: replace an OST with the same index ========================================================== 22:41:14 (1788748874)
start mds service on oleg346-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep  6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg346-server'
oleg346-server: oleg346-server.virtnet: executing load_modules_local
oleg346-server: kmemleak disabled
oleg346-server: Loading modules from /home/green/git/lustre-release/lustre
oleg346-server: detected 4 online CPUs by sysfs
oleg346-server: MODOPTS_LIBCFS=
oleg346-server: Force libcfs to create 2 CPU partitions
oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg346-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg346-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg346-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-OST0000
oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg346-client.virtnet:  -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 808 (time 1788748948.82 total 10.01 last 80.75)
 - open/close 1575 (time 1788748958.82 total 20.01 last 76.65)
 - open/close 2342 (time 1788748968.82 total 30.01 last 76.69)
 - open/close 3134 (time 1788748978.82 total 40.01 last 79.19)
 - open/close 3938 (time 1788748988.84 total 50.03 last 80.25)
 - open/close 4831 (time 1788748998.85 total 60.04 last 89.22)
 - open/close 5664 (time 1788749008.86 total 70.05 last 83.21)
 - open/close 6475 (time 1788749018.87 total 80.06 last 81.08)
 - open/close 7276 (time 1788749028.87 total 90.06 last 80.05)
 - open/close 8111 (time 1788749038.89 total 100.08 last 83.39)
 - open/close 8912 (time 1788749048.90 total 110.09 last 80.01)
 - open/close 9664 (time 1788749058.90 total 120.09 last 75.17)
total: 10000 open/close in 124.44 seconds: 80.36 ops/second
 - unlinked 0 (time 1788749066 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 695 (time 1788749154.04 total 10.01 last 69.43)
 - open/close 1480 (time 1788749164.05 total 20.02 last 78.45)
 - open/close 2370 (time 1788749174.05 total 30.02 last 88.97)
 - open/close 3239 (time 1788749184.05 total 40.02 last 86.89)
 - open/close 4062 (time 1788749194.06 total 50.03 last 82.25)
 - open/close 4871 (time 1788749204.07 total 60.04 last 80.85)
 - open/close 5675 (time 1788749214.07 total 70.04 last 80.40)
 - open/close 6494 (time 1788749224.09 total 80.06 last 81.70)
 - open/close 7299 (time 1788749234.09 total 90.06 last 80.49)
 - open/close 8054 (time 1788749244.10 total 100.07 last 75.48)
 - open/close 8900 (time 1788749254.11 total 110.07 last 84.52)
 - open/close 9761 (time 1788749264.11 total 120.08 last 86.09)
total: 10000 open/close in 123.12 seconds: 81.22 ops/second
 - unlinked 0 (time 1788749270 ; total 0 ; last 0)
total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second
 - open/close 867 (time 1788749351.42 total 10.01 last 86.65)
 - open/close 1715 (time 1788749361.43 total 20.02 last 84.71)
 - open/close 2530 (time 1788749371.44 total 30.02 last 81.46)
 - open/close 3334 (time 1788749381.45 total 40.03 last 80.32)
 - open/close 4137 (time 1788749391.45 total 50.03 last 80.30)
 - open/close 4899 (time 1788749401.45 total 60.04 last 76.16)
 - open/close 5683 (time 1788749411.46 total 70.04 last 78.35)
 - open/close 6448 (time 1788749421.47 total 80.05 last 76.43)
 - open/close 7254 (time 1788749431.48 total 90.06 last 80.50)
 - open/close 8120 (time 1788749441.49 total 100.07 last 86.54)
 - open/close 8935 (time 1788749451.50 total 110.08 last 81.43)
 - open/close 9800 (time 1788749461.50 total 120.08 last 86.49)
total: 10000 open/close in 122.21 seconds: 81.82 ops/second
 - unlinked 0 (time 1788749466 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 441 (time 1788749567.30 total 10.02 last 44.02)
 - open/close 893 (time 1788749577.32 total 20.04 last 45.10)
 - open/close 1547 (time 1788749587.33 total 30.05 last 65.33)
 - open/close 2182 (time 1788749597.34 total 40.06 last 63.46)
 - open/close 2814 (time 1788749607.35 total 50.07 last 63.12)
 - open/close 3408 (time 1788749617.38 total 60.10 last 59.22)
 - open/close 4106 (time 1788749627.39 total 70.11 last 69.72)
 - open/close 4701 (time 1788749637.39 total 80.12 last 59.49)
 - open/close 5295 (time 1788749647.40 total 90.12 last 59.35)
 - open/close 5930 (time 1788749657.42 total 100.15 last 63.37)
 - open/close 6563 (time 1788749667.44 total 110.16 last 63.23)
 - open/close 7146 (time 1788749677.45 total 120.17 last 58.21)
 - open/close 7776 (time 1788749687.46 total 130.18 last 62.92)
 - open/close 8347 (time 1788749697.47 total 140.19 last 57.06)
 - open/close 8984 (time 1788749707.47 total 150.19 last 63.70)
 - open/close 9645 (time 1788749717.48 total 160.20 last 66.04)
total: 10000 open/close in 165.40 seconds: 60.46 ops/second
 - unlinked 0 (time 1788749726 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 650 (time 1788749831.66 total 10.01 last 64.93)
 - open/close 1223 (time 1788749841.66 total 20.01 last 57.28)
 - open/close 1813 (time 1788749851.66 total 30.02 last 58.96)
 - open/close 2413 (time 1788749861.67 total 40.02 last 60.00)
 - open/close 2987 (time 1788749871.71 total 50.06 last 57.15)
 - open/close 3530 (time 1788749881.73 total 60.08 last 54.20)
 - open/close 4229 (time 1788749891.74 total 70.09 last 69.83)
 - open/close 4881 (time 1788749901.76 total 80.11 last 65.07)
 - open/close 5520 (time 1788749911.76 total 90.11 last 63.89)
 - open/close 6191 (time 1788749921.78 total 100.13 last 66.99)
 - open/close 6766 (time 1788749931.78 total 110.14 last 57.44)
 - open/close 7385 (time 1788749941.79 total 120.14 last 61.87)
 - open/close 8106 (time 1788749951.80 total 130.15 last 72.05)
 - open/close 8750 (time 1788749961.81 total 140.17 last 64.29)
 - open/close 9367 (time 1788749971.82 total 150.17 last 61.68)
 - open/close 9867 (time 1788749981.82 total 160.18 last 49.97)
total: 10000 open/close in 162.94 seconds: 61.37 ops/second
 - unlinked 0 (time 1788749990 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 705 (time 1788750105.20 total 10.02 last 70.35)
 - open/close 1369 (time 1788750115.20 total 20.02 last 66.40)
 - open/close 2055 (time 1788750125.22 total 30.04 last 68.47)
 - open/close 2742 (time 1788750135.22 total 40.04 last 68.68)
 - open/close 3466 (time 1788750145.24 total 50.06 last 72.31)
 - open/close 4120 (time 1788750155.26 total 60.08 last 65.26)
 - open/close 4778 (time 1788750165.27 total 70.09 last 65.71)
 - open/close 5431 (time 1788750175.27 total 80.09 last 65.28)
 - open/close 6089 (time 1788750185.28 total 90.10 last 65.78)
 - open/close 6715 (time 1788750195.30 total 100.12 last 62.46)
 - open/close 7350 (time 1788750205.31 total 110.13 last 63.46)
 - open/close 8062 (time 1788750215.32 total 120.14 last 71.10)
 - open/close 8686 (time 1788750225.32 total 130.14 last 62.40)
 - open/close 9369 (time 1788750235.33 total 140.15 last 68.23)
total: 10000 open/close in 149.05 seconds: 67.09 ops/second
 - unlinked 0 (time 1788750248 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 600 (time 1788750356.50 total 10.01 last 59.97)
 - open/close 1179 (time 1788750366.50 total 20.01 last 57.89)
 - open/close 1809 (time 1788750376.51 total 30.02 last 62.94)
 - open/close 2449 (time 1788750386.53 total 40.03 last 63.90)
 - open/close 3086 (time 1788750396.55 total 50.06 last 63.56)
 - open/close 3666 (time 1788750406.56 total 60.06 last 57.98)
 - open/close 4316 (time 1788750416.56 total 70.06 last 64.99)
 - open/close 4903 (time 1788750426.56 total 80.06 last 58.69)
 - open/close 5532 (time 1788750436.56 total 90.06 last 62.89)
 - open/close 6111 (time 1788750446.57 total 100.07 last 57.86)
 - open/close 6748 (time 1788750456.57 total 110.07 last 63.69)
 - open/close 7351 (time 1788750466.57 total 120.07 last 60.29)
 - open/close 7977 (time 1788750476.57 total 130.08 last 62.58)
 - open/close 8620 (time 1788750486.58 total 140.08 last 64.28)
 - open/close 9333 (time 1788750496.59 total 150.09 last 71.24)
total: 10000 open/close in 159.89 seconds: 62.54 ops/second
 - unlinked 0 (time 1788750510 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 692 (time 1788750622.81 total 10.01 last 69.15)
 - open/close 1290 (time 1788750632.82 total 20.02 last 59.71)
 - open/close 1961 (time 1788750642.83 total 30.02 last 67.08)
 - open/close 2645 (time 1788750652.83 total 40.03 last 68.37)
 - open/close 3331 (time 1788750662.84 total 50.04 last 68.55)
 - open/close 4053 (time 1788750672.84 total 60.04 last 72.18)
 - open/close 4737 (time 1788750682.85 total 70.05 last 68.32)
 - open/close 5386 (time 1788750692.86 total 80.06 last 64.88)
 - open/close 6028 (time 1788750702.86 total 90.06 last 64.19)
 - open/close 6778 (time 1788750712.86 total 100.06 last 74.96)
 - open/close 7407 (time 1788750722.87 total 110.07 last 62.86)
 - open/close 8098 (time 1788750732.87 total 120.07 last 69.09)
 - open/close 8774 (time 1788750742.87 total 130.07 last 67.58)
 - open/close 9552 (time 1788750752.89 total 140.08 last 77.71)
total: 10000 open/close in 146.76 seconds: 68.14 ops/second
 - unlinked 0 (time 1788750764 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 733 (time 1788750869.26 total 10.01 last 73.25)
 - open/close 1474 (time 1788750879.27 total 20.01 last 74.09)
 - open/close 2188 (time 1788750889.28 total 30.02 last 71.33)
 - open/close 2853 (time 1788750899.28 total 40.02 last 66.46)
 - open/close 3468 (time 1788750909.30 total 50.04 last 61.42)
 - open/close 4100 (time 1788750919.30 total 60.04 last 63.17)
 - open/close 4736 (time 1788750929.30 total 70.04 last 63.59)
 - open/close 5415 (time 1788750939.31 total 80.05 last 67.87)
 - open/close 6073 (time 1788750949.32 total 90.06 last 65.74)
 - open/close 6765 (time 1788750959.33 total 100.08 last 69.07)
 - open/close 7435 (time 1788750969.34 total 110.08 last 66.97)
 - open/close 8124 (time 1788750979.35 total 120.09 last 68.85)
 - open/close 8816 (time 1788750989.35 total 130.09 last 69.20)
 - open/close 9515 (time 1788750999.35 total 140.09 last 69.86)
total: 10000 open/close in 147.28 seconds: 67.90 ops/second
 - unlinked 0 (time 1788751010 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 688 (time 1788751114.00 total 10.00 last 68.79)
 - open/close 1366 (time 1788751124.01 total 20.01 last 67.75)
 - open/close 2091 (time 1788751134.01 total 30.01 last 72.47)
 - open/close 2776 (time 1788751144.01 total 40.02 last 68.47)
 - open/close 3457 (time 1788751154.02 total 50.02 last 68.07)
 - open/close 4217 (time 1788751164.03 total 60.03 last 75.94)
 - open/close 4871 (time 1788751174.03 total 70.03 last 65.38)
 - open/close 5576 (time 1788751184.04 total 80.04 last 70.44)
 - open/close 6268 (time 1788751194.04 total 90.05 last 69.17)
 - open/close 6962 (time 1788751204.06 total 100.06 last 69.28)
 - open/close 7616 (time 1788751214.13 total 110.13 last 64.94)
 - open/close 8245 (time 1788751224.13 total 120.14 last 62.88)
 - open/close 8963 (time 1788751234.14 total 130.14 last 71.77)
 - open/close 9682 (time 1788751244.15 total 140.15 last 71.83)
total: 10000 open/close in 144.64 seconds: 69.14 ops/second
 - unlinked 0 (time 1788751251 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg346-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg346-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg346-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-OST0000
oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg346-client.virtnet:  -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg346-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg346-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg346-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-OST0000
oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg346-client.virtnet:  -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg346-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg346-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server
stop mds service on oleg346-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server
unloading modules via unload_modules_local on: 'oleg346-server'
oleg346-server: oleg346-server.virtnet: executing unload_modules_local
oleg346-server: kmemleak disabled
modules unloaded.
pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2
pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2