== conf-sanity test 69: replace an OST with the same index ========================================================== 13:38:28 (1789061908)
start mds service on oleg305-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep 10 02:33 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg305-server'
oleg305-server: oleg305-server.virtnet: executing load_modules_local
oleg305-server: kmemleak disabled
oleg305-server: Loading modules from /home/green/git/lustre-release/lustre
oleg305-server: detected 4 online CPUs by sysfs
oleg305-server: MODOPTS_LIBCFS=
oleg305-server: Force libcfs to create 2 CPU partitions
oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg305-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg305-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg305-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1
Started lustre-OST0000
oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg305-client.virtnet:  -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 690 (time 1789061989.19 total 10.01 last 68.96)
 - open/close 1355 (time 1789061999.20 total 20.01 last 66.46)
 - open/close 2028 (time 1789062009.21 total 30.02 last 67.24)
 - open/close 2707 (time 1789062019.21 total 40.02 last 67.88)
 - open/close 3360 (time 1789062029.22 total 50.03 last 65.27)
 - open/close 4051 (time 1789062039.22 total 60.03 last 69.09)
 - open/close 4729 (time 1789062049.22 total 70.04 last 67.76)
 - open/close 5403 (time 1789062059.22 total 80.04 last 67.39)
 - open/close 6064 (time 1789062069.24 total 90.06 last 65.96)
 - open/close 6714 (time 1789062079.25 total 100.07 last 64.95)
 - open/close 7422 (time 1789062089.25 total 110.07 last 70.80)
 - open/close 8102 (time 1789062099.27 total 120.08 last 67.91)
 - open/close 8782 (time 1789062109.27 total 130.09 last 67.96)
 - open/close 9473 (time 1789062119.28 total 140.09 last 69.06)
total: 10000 open/close in 147.24 seconds: 67.92 ops/second
 - unlinked 0 (time 1789062130 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 641 (time 1789062237.80 total 10.01 last 64.04)
 - open/close 1317 (time 1789062247.81 total 20.01 last 67.59)
 - open/close 1976 (time 1789062257.81 total 30.02 last 65.85)
 - open/close 2704 (time 1789062267.82 total 40.03 last 72.74)
 - open/close 3386 (time 1789062277.84 total 50.05 last 68.08)
 - open/close 3994 (time 1789062287.85 total 60.06 last 60.74)
 - open/close 4601 (time 1789062297.87 total 70.07 last 60.60)
 - open/close 5258 (time 1789062307.88 total 80.09 last 65.58)
 - open/close 5906 (time 1789062317.89 total 90.09 last 64.77)
 - open/close 6481 (time 1789062327.90 total 100.10 last 57.45)
 - open/close 7032 (time 1789062337.91 total 110.11 last 55.05)
 - open/close 7639 (time 1789062347.92 total 120.12 last 60.64)
 - open/close 8173 (time 1789062357.93 total 130.13 last 53.33)
 - open/close 8761 (time 1789062367.94 total 140.15 last 58.72)
 - open/close 9218 (time 1789062377.95 total 150.16 last 45.65)
 - open/close 9835 (time 1789062387.95 total 160.16 last 61.69)
total: 10000 open/close in 163.54 seconds: 61.15 ops/second
 - unlinked 0 (time 1789062397 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 635 (time 1789062508.89 total 10.01 last 63.45)
 - open/close 1272 (time 1789062518.91 total 20.03 last 63.53)
 - open/close 1883 (time 1789062528.92 total 30.04 last 61.05)
 - open/close 2472 (time 1789062538.95 total 40.07 last 58.76)
 - open/close 3168 (time 1789062548.95 total 50.07 last 69.59)
 - open/close 3834 (time 1789062558.96 total 60.08 last 66.52)
 - open/close 4493 (time 1789062568.96 total 70.08 last 65.87)
 - open/close 5142 (time 1789062578.96 total 80.09 last 64.89)
 - open/close 5783 (time 1789062588.97 total 90.09 last 64.06)
 - open/close 6410 (time 1789062598.98 total 100.10 last 62.67)
 - open/close 6982 (time 1789062608.98 total 110.10 last 57.17)
 - open/close 7533 (time 1789062619.00 total 120.12 last 55.02)
 - open/close 8042 (time 1789062629.01 total 130.13 last 50.83)
 - open/close 8652 (time 1789062639.01 total 140.13 last 60.99)
 - open/close 9252 (time 1789062649.02 total 150.15 last 59.92)
 - open/close 9889 (time 1789062659.03 total 160.15 last 63.68)
total: 10000 open/close in 161.72 seconds: 61.83 ops/second
 - unlinked 0 (time 1789062665 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 665 (time 1789062774.96 total 10.00 last 66.49)
 - open/close 1351 (time 1789062784.98 total 20.02 last 68.47)
 - open/close 2006 (time 1789062794.99 total 30.03 last 65.45)
 - open/close 2684 (time 1789062804.99 total 40.03 last 67.77)
 - open/close 3318 (time 1789062815.01 total 50.05 last 63.26)
 - open/close 3919 (time 1789062825.01 total 60.05 last 60.10)
 - open/close 4576 (time 1789062835.03 total 70.07 last 65.55)
 - open/close 5244 (time 1789062845.04 total 80.08 last 66.75)
 - open/close 5897 (time 1789062855.05 total 90.09 last 65.26)
 - open/close 6521 (time 1789062865.05 total 100.10 last 62.35)
 - open/close 7163 (time 1789062875.07 total 110.11 last 64.11)
 - open/close 7724 (time 1789062885.07 total 120.11 last 56.08)
 - open/close 8378 (time 1789062895.07 total 130.11 last 65.39)
 - open/close 9061 (time 1789062905.07 total 140.12 last 68.29)
 - open/close 9812 (time 1789062915.08 total 150.12 last 75.05)
total: 10000 open/close in 153.32 seconds: 65.22 ops/second
 - unlinked 0 (time 1789062922 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 620 (time 1789063044.12 total 10.01 last 61.94)
 - open/close 1135 (time 1789063054.16 total 20.05 last 51.31)
 - open/close 1690 (time 1789063064.16 total 30.05 last 55.49)
 - open/close 2325 (time 1789063074.18 total 40.07 last 63.39)
 - open/close 2884 (time 1789063084.18 total 50.07 last 55.90)
 - open/close 3518 (time 1789063094.19 total 60.08 last 63.35)
 - open/close 4257 (time 1789063104.19 total 70.08 last 73.85)
 - open/close 4956 (time 1789063114.19 total 80.08 last 69.89)
 - open/close 5563 (time 1789063124.20 total 90.09 last 60.64)
 - open/close 6104 (time 1789063134.21 total 100.10 last 54.04)
 - open/close 6824 (time 1789063144.24 total 110.13 last 71.82)
 - open/close 7469 (time 1789063154.24 total 120.13 last 64.49)
 - open/close 8183 (time 1789063164.25 total 130.14 last 71.32)
 - open/close 8848 (time 1789063174.26 total 140.15 last 66.42)
 - open/close 9578 (time 1789063184.27 total 150.16 last 72.95)
total: 10000 open/close in 157.40 seconds: 63.53 ops/second
 - unlinked 0 (time 1789063197 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 606 (time 1789063315.90 total 10.00 last 60.58)
 - open/close 1264 (time 1789063325.91 total 20.01 last 65.73)
 - open/close 1950 (time 1789063335.91 total 30.02 last 68.58)
 - open/close 2621 (time 1789063345.92 total 40.03 last 66.99)
 - open/close 3276 (time 1789063355.93 total 50.04 last 65.45)
 - open/close 3940 (time 1789063365.94 total 60.05 last 66.34)
 - open/close 4617 (time 1789063375.96 total 70.06 last 67.59)
 - open/close 5327 (time 1789063385.96 total 80.07 last 70.99)
 - open/close 6022 (time 1789063395.97 total 90.08 last 69.42)
 - open/close 6646 (time 1789063405.97 total 100.08 last 62.40)
 - open/close 7206 (time 1789063415.99 total 110.09 last 55.92)
 - open/close 7818 (time 1789063426.00 total 120.11 last 61.12)
 - open/close 8307 (time 1789063436.00 total 130.11 last 48.89)
 - open/close 8919 (time 1789063446.01 total 140.11 last 61.16)
 - open/close 9605 (time 1789063456.01 total 150.12 last 68.57)
total: 10000 open/close in 155.71 seconds: 64.22 ops/second
 - unlinked 0 (time 1789063466 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 790 (time 1789063570.83 total 10.01 last 78.93)
 - open/close 1352 (time 1789063580.84 total 20.02 last 56.15)
 - open/close 2028 (time 1789063590.84 total 30.02 last 67.60)
 - open/close 2657 (time 1789063600.85 total 40.02 last 62.86)
 - open/close 3300 (time 1789063610.88 total 50.05 last 64.11)
 - open/close 3878 (time 1789063620.89 total 60.07 last 57.70)
 - open/close 4593 (time 1789063630.90 total 70.08 last 71.42)
 - open/close 5224 (time 1789063640.91 total 80.08 last 63.08)
 - open/close 5951 (time 1789063650.91 total 90.09 last 72.69)
 - open/close 6632 (time 1789063660.91 total 100.09 last 68.09)
 - open/close 7374 (time 1789063670.93 total 110.10 last 74.07)
 - open/close 8159 (time 1789063680.93 total 120.11 last 78.48)
 - open/close 8804 (time 1789063690.93 total 130.11 last 64.49)
 - open/close 9445 (time 1789063700.93 total 140.11 last 64.10)
total: 10000 open/close in 148.87 seconds: 67.17 ops/second
 - unlinked 0 (time 1789063714 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 667 (time 1789063815.77 total 10.01 last 66.67)
 - open/close 1316 (time 1789063825.77 total 20.01 last 64.85)
 - open/close 1867 (time 1789063835.78 total 30.02 last 55.08)
 - open/close 2592 (time 1789063845.79 total 40.03 last 72.40)
 - open/close 3246 (time 1789063855.79 total 50.03 last 65.38)
 - open/close 3992 (time 1789063865.80 total 60.04 last 74.54)
 - open/close 4747 (time 1789063875.81 total 70.05 last 75.45)
 - open/close 5419 (time 1789063885.84 total 80.08 last 67.00)
 - open/close 6121 (time 1789063895.84 total 90.08 last 70.17)
 - open/close 6753 (time 1789063905.85 total 100.09 last 63.14)
 - open/close 7440 (time 1789063915.87 total 110.10 last 68.61)
 - open/close 8231 (time 1789063925.87 total 120.11 last 79.07)
 - open/close 8989 (time 1789063935.87 total 130.11 last 75.77)
 - open/close 9614 (time 1789063945.88 total 140.12 last 62.48)
total: 10000 open/close in 145.31 seconds: 68.82 ops/second
 - unlinked 0 (time 1789063955 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 675 (time 1789064058.35 total 10.01 last 67.43)
 - open/close 1342 (time 1789064068.35 total 20.01 last 66.69)
 - open/close 1982 (time 1789064078.35 total 30.01 last 63.99)
 - open/close 2564 (time 1789064088.36 total 40.02 last 58.13)
 - open/close 3176 (time 1789064098.36 total 50.03 last 61.19)
 - open/close 3925 (time 1789064108.37 total 60.03 last 74.85)
 - open/close 4601 (time 1789064118.37 total 70.03 last 67.58)
 - open/close 5066 (time 1789064128.40 total 80.06 last 46.39)
 - open/close 5545 (time 1789064138.41 total 90.07 last 47.85)
 - open/close 6161 (time 1789064148.41 total 100.07 last 61.57)
 - open/close 6760 (time 1789064158.42 total 110.08 last 59.87)
 - open/close 7387 (time 1789064168.44 total 120.10 last 62.56)
 - open/close 7991 (time 1789064178.46 total 130.12 last 60.29)
 - open/close 8667 (time 1789064188.46 total 140.12 last 67.59)
 - open/close 9359 (time 1789064198.47 total 150.13 last 69.15)
total: 10000 open/close in 159.87 seconds: 62.55 ops/second
 - unlinked 0 (time 1789064213 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 780 (time 1789064318.42 total 10.01 last 77.89)
 - open/close 1397 (time 1789064328.44 total 20.03 last 61.62)
 - open/close 2109 (time 1789064338.44 total 30.03 last 71.17)
 - open/close 2807 (time 1789064348.45 total 40.04 last 69.73)
 - open/close 3576 (time 1789064358.46 total 50.05 last 76.79)
 - open/close 4263 (time 1789064368.47 total 60.06 last 68.66)
 - open/close 5026 (time 1789064378.48 total 70.07 last 76.24)
 - open/close 5764 (time 1789064388.49 total 80.07 last 73.75)
 - open/close 6467 (time 1789064398.49 total 90.08 last 70.28)
 - open/close 7190 (time 1789064408.50 total 100.09 last 72.19)
 - open/close 7885 (time 1789064418.51 total 110.10 last 69.44)
 - open/close 8511 (time 1789064428.52 total 120.11 last 62.53)
 - open/close 9123 (time 1789064438.52 total 130.11 last 61.19)
 - open/close 9772 (time 1789064448.52 total 140.11 last 64.90)
total: 10000 open/close in 143.20 seconds: 69.83 ops/second
 - unlinked 0 (time 1789064456 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg305-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg305-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.105@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.105@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg305-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1
Started lustre-OST0000
oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg305-server: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg305-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg305-client.virtnet:  -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre
On OST0, 18670 used inodes rc=0
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg305-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg305-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.105@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.105@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.105@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.105@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.105@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg305-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1
Started lustre-OST0000
oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg305-server: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg305-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg305-client.virtnet:  -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg305-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg305-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server
stop mds service on oleg305-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server
unloading modules via unload_modules_local on: 'oleg305-server'
oleg305-server: oleg305-server.virtnet: executing unload_modules_local
oleg305-server: kmemleak disabled
modules unloaded.
pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2
pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2
pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2