== conf-sanity test 69: replace an OST with the same index ========================================================== 22:12:24 (1788833544)
start mds service on oleg139-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  7 17:47 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg139-server'
oleg139-server: oleg139-server.virtnet: executing load_modules_local
oleg139-server: kmemleak disabled
oleg139-server: Loading modules from /home/green/git/lustre-release/lustre
oleg139-server: detected 4 online CPUs by sysfs
oleg139-server: MODOPTS_LIBCFS=
oleg139-server: Force libcfs to create 2 CPU partitions
oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg139-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 580 (time 1788833646.10 total 10.01 last 57.95)
 - open/close 1132 (time 1788833656.10 total 20.01 last 55.18)
 - open/close 1691 (time 1788833666.13 total 30.04 last 55.74)
 - open/close 2302 (time 1788833676.15 total 40.06 last 61.00)
 - open/close 2852 (time 1788833686.16 total 50.07 last 54.93)
 - open/close 3437 (time 1788833696.16 total 60.07 last 58.49)
 - open/close 3994 (time 1788833706.17 total 70.08 last 55.64)
 - open/close 4580 (time 1788833716.18 total 80.10 last 58.53)
 - open/close 5107 (time 1788833726.18 total 90.10 last 52.69)
 - open/close 5624 (time 1788833736.19 total 100.10 last 51.66)
 - open/close 6234 (time 1788833746.21 total 110.12 last 60.90)
 - open/close 6854 (time 1788833756.22 total 120.14 last 61.91)
 - open/close 7494 (time 1788833766.23 total 130.15 last 63.92)
 - open/close 8157 (time 1788833776.24 total 140.16 last 66.23)
 - open/close 8795 (time 1788833786.25 total 150.16 last 63.77)
 - open/close 9356 (time 1788833796.26 total 160.18 last 56.03)
 - open/close 9814 (time 1788833806.27 total 170.19 last 45.75)
total: 10000 open/close in 173.40 seconds: 57.67 ops/second
 - unlinked 0 (time 1788833815 ; total 0 ; last 0)
total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second
 - open/close 566 (time 1788833947.61 total 10.00 last 56.58)
 - open/close 1255 (time 1788833957.62 total 20.01 last 68.84)
 - open/close 1968 (time 1788833967.62 total 30.02 last 71.26)
 - open/close 2582 (time 1788833977.63 total 40.03 last 61.35)
 - open/close 3206 (time 1788833987.65 total 50.05 last 62.28)
 - open/close 3861 (time 1788833997.65 total 60.05 last 65.49)
 - open/close 4369 (time 1788834007.66 total 70.05 last 50.77)
 - open/close 4845 (time 1788834017.66 total 80.06 last 47.58)
 - open/close 5309 (time 1788834027.67 total 90.07 last 46.36)
 - open/close 6012 (time 1788834037.68 total 100.07 last 70.24)
 - open/close 6623 (time 1788834047.68 total 110.08 last 61.09)
 - open/close 7265 (time 1788834057.72 total 120.11 last 63.98)
 - open/close 7794 (time 1788834067.73 total 130.12 last 52.83)
 - open/close 8429 (time 1788834077.73 total 140.12 last 63.50)
 - open/close 9070 (time 1788834087.73 total 150.13 last 64.08)
 - open/close 9660 (time 1788834097.75 total 160.14 last 58.91)
total: 10000 open/close in 165.33 seconds: 60.48 ops/second
 - unlinked 0 (time 1788834108 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 653 (time 1788834223.16 total 10.00 last 65.29)
 - open/close 1350 (time 1788834233.18 total 20.02 last 69.56)
 - open/close 1988 (time 1788834243.19 total 30.03 last 63.76)
 - open/close 2594 (time 1788834253.19 total 40.03 last 60.58)
 - open/close 3174 (time 1788834263.20 total 50.04 last 57.93)
 - open/close 3776 (time 1788834273.21 total 60.05 last 60.15)
 - open/close 4447 (time 1788834283.21 total 70.06 last 67.08)
 - open/close 5021 (time 1788834293.23 total 80.07 last 57.31)
 - open/close 5688 (time 1788834303.23 total 90.08 last 66.68)
 - open/close 6254 (time 1788834313.24 total 100.08 last 56.57)
 - open/close 6777 (time 1788834323.25 total 110.09 last 52.27)
 - open/close 7351 (time 1788834333.25 total 120.09 last 57.37)
 - open/close 7988 (time 1788834343.25 total 130.09 last 63.70)
 - open/close 8659 (time 1788834353.26 total 140.10 last 67.02)
 - open/close 9218 (time 1788834363.28 total 150.12 last 55.79)
 - open/close 9870 (time 1788834373.29 total 160.13 last 65.16)
total: 10000 open/close in 162.16 seconds: 61.67 ops/second
 - unlinked 0 (time 1788834380 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 687 (time 1788834498.85 total 10.01 last 68.65)
 - open/close 1306 (time 1788834508.86 total 20.01 last 61.87)
 - open/close 1956 (time 1788834518.88 total 30.04 last 64.84)
 - open/close 2457 (time 1788834528.89 total 40.05 last 50.05)
 - open/close 3079 (time 1788834538.91 total 50.06 last 62.09)
 - open/close 3663 (time 1788834548.91 total 60.07 last 58.38)
 - open/close 4220 (time 1788834558.92 total 70.08 last 55.65)
 - open/close 4826 (time 1788834568.92 total 80.08 last 60.57)
 - open/close 5348 (time 1788834578.94 total 90.09 last 52.14)
 - open/close 6114 (time 1788834588.94 total 100.09 last 76.60)
 - open/close 6786 (time 1788834598.94 total 110.10 last 67.18)
 - open/close 7424 (time 1788834608.94 total 120.10 last 63.80)
 - open/close 8123 (time 1788834618.95 total 130.11 last 69.83)
 - open/close 8811 (time 1788834628.96 total 140.11 last 68.75)
 - open/close 9471 (time 1788834638.96 total 150.12 last 65.98)
total: 10000 open/close in 156.75 seconds: 63.79 ops/second
 - unlinked 0 (time 1788834650 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 506 (time 1788834756.43 total 10.00 last 50.59)
 - open/close 1228 (time 1788834766.43 total 20.00 last 72.20)
 - open/close 1904 (time 1788834776.45 total 30.02 last 67.46)
 - open/close 2532 (time 1788834786.46 total 40.03 last 62.77)
 - open/close 3207 (time 1788834796.47 total 50.04 last 67.43)
 - open/close 3911 (time 1788834806.47 total 60.04 last 70.40)
 - open/close 4560 (time 1788834816.50 total 70.07 last 64.68)
 - open/close 5227 (time 1788834826.51 total 80.09 last 66.62)
 - open/close 5921 (time 1788834836.52 total 90.09 last 69.39)
 - open/close 6583 (time 1788834846.52 total 100.09 last 66.19)
 - open/close 7168 (time 1788834856.52 total 110.10 last 58.45)
 - open/close 7815 (time 1788834866.53 total 120.10 last 64.70)
 - open/close 8636 (time 1788834876.53 total 130.11 last 82.02)
 - open/close 9374 (time 1788834886.54 total 140.12 last 73.74)
total: 10000 open/close in 149.78 seconds: 66.76 ops/second
 - unlinked 0 (time 1788834901 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 574 (time 1788835013.55 total 10.01 last 57.36)
 - open/close 1202 (time 1788835023.56 total 20.01 last 62.75)
 - open/close 1696 (time 1788835033.57 total 30.02 last 49.36)
 - open/close 2287 (time 1788835043.58 total 40.03 last 59.05)
 - open/close 3072 (time 1788835053.58 total 50.04 last 78.47)
 - open/close 3802 (time 1788835063.58 total 60.04 last 72.99)
 - open/close 4442 (time 1788835073.60 total 70.05 last 63.89)
 - open/close 5175 (time 1788835083.61 total 80.07 last 73.21)
 - open/close 5694 (time 1788835093.61 total 90.07 last 51.89)
 - open/close 6385 (time 1788835103.65 total 100.11 last 68.83)
 - open/close 7138 (time 1788835113.67 total 110.12 last 75.19)
 - open/close 7733 (time 1788835123.67 total 120.13 last 59.47)
 - open/close 8338 (time 1788835133.67 total 130.13 last 60.49)
 - open/close 9028 (time 1788835143.68 total 140.13 last 68.97)
 - open/close 9714 (time 1788835153.69 total 150.14 last 68.54)
total: 10000 open/close in 154.47 seconds: 64.74 ops/second
 - unlinked 0 (time 1788835162 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 529 (time 1788835268.81 total 10.01 last 52.87)
 - open/close 1209 (time 1788835278.85 total 20.04 last 67.76)
 - open/close 1835 (time 1788835288.85 total 30.04 last 62.59)
 - open/close 2497 (time 1788835298.87 total 40.07 last 66.03)
 - open/close 3154 (time 1788835308.88 total 50.07 last 65.66)
 - open/close 3792 (time 1788835318.88 total 60.08 last 63.77)
 - open/close 4404 (time 1788835328.89 total 70.08 last 61.17)
 - open/close 5066 (time 1788835338.89 total 80.09 last 66.17)
 - open/close 5734 (time 1788835348.91 total 90.10 last 66.70)
 - open/close 6388 (time 1788835358.91 total 100.10 last 65.39)
 - open/close 7030 (time 1788835368.92 total 110.11 last 64.16)
 - open/close 7701 (time 1788835378.93 total 120.12 last 67.03)
 - open/close 8355 (time 1788835388.96 total 130.15 last 65.20)
 - open/close 9138 (time 1788835398.97 total 140.16 last 78.23)
 - open/close 9832 (time 1788835408.98 total 150.17 last 69.33)
total: 10000 open/close in 153.26 seconds: 65.25 ops/second
 - unlinked 0 (time 1788835417 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 370 (time 1788835533.24 total 10.00 last 37.00)
 - open/close 967 (time 1788835543.27 total 20.03 last 59.52)
 - open/close 1598 (time 1788835553.28 total 30.03 last 63.08)
 - open/close 2268 (time 1788835563.28 total 40.04 last 66.95)
 - open/close 2936 (time 1788835573.29 total 50.05 last 66.73)
 - open/close 3621 (time 1788835583.30 total 60.05 last 68.48)
 - open/close 4315 (time 1788835593.31 total 70.07 last 69.32)
 - open/close 4895 (time 1788835603.31 total 80.07 last 57.98)
 - open/close 5489 (time 1788835613.33 total 90.09 last 59.28)
 - open/close 6127 (time 1788835623.35 total 100.11 last 63.67)
 - open/close 6760 (time 1788835633.36 total 110.12 last 63.24)
 - open/close 7328 (time 1788835643.38 total 120.14 last 56.67)
 - open/close 7822 (time 1788835653.39 total 130.15 last 49.36)
 - open/close 8327 (time 1788835663.40 total 140.16 last 50.44)
 - open/close 9050 (time 1788835673.41 total 150.17 last 72.24)
 - open/close 9584 (time 1788835683.50 total 160.26 last 52.92)
total: 10000 open/close in 167.46 seconds: 59.72 ops/second
 - unlinked 0 (time 1788835695 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 605 (time 1788835809.91 total 10.00 last 60.48)
 - open/close 1186 (time 1788835819.91 total 20.01 last 58.08)
 - open/close 1717 (time 1788835829.92 total 30.01 last 53.05)
 - open/close 2277 (time 1788835839.92 total 40.02 last 55.97)
 - open/close 2889 (time 1788835849.94 total 50.03 last 61.12)
 - open/close 3504 (time 1788835859.94 total 60.03 last 61.49)
 - open/close 4084 (time 1788835869.95 total 70.04 last 57.95)
 - open/close 4658 (time 1788835879.95 total 80.05 last 57.37)
 - open/close 5135 (time 1788835889.97 total 90.06 last 47.63)
 - open/close 5720 (time 1788835899.97 total 100.07 last 58.47)
 - open/close 6393 (time 1788835909.99 total 110.09 last 67.17)
 - open/close 7011 (time 1788835919.99 total 120.09 last 61.78)
 - open/close 7653 (time 1788835930.00 total 130.10 last 64.14)
 - open/close 8299 (time 1788835940.00 total 140.10 last 64.59)
 - open/close 8916 (time 1788835950.00 total 150.10 last 61.70)
 - open/close 9482 (time 1788835960.01 total 160.11 last 56.56)
total: 10000 open/close in 168.75 seconds: 59.26 ops/second
 - unlinked 0 (time 1788835973 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 670 (time 1788836083.81 total 10.00 last 66.98)
 - open/close 1306 (time 1788836093.83 total 20.02 last 63.47)
 - open/close 2054 (time 1788836103.83 total 30.02 last 74.79)
 - open/close 2680 (time 1788836113.85 total 40.04 last 62.51)
 - open/close 3400 (time 1788836123.85 total 50.04 last 71.98)
 - open/close 4086 (time 1788836133.85 total 60.05 last 68.58)
 - open/close 4755 (time 1788836143.87 total 70.07 last 66.77)
 - open/close 5502 (time 1788836153.88 total 80.07 last 74.66)
 - open/close 6280 (time 1788836163.89 total 90.08 last 77.70)
 - open/close 7017 (time 1788836173.90 total 100.09 last 73.64)
 - open/close 7623 (time 1788836183.91 total 110.10 last 60.55)
 - open/close 8313 (time 1788836193.91 total 120.11 last 68.94)
 - open/close 9048 (time 1788836203.92 total 130.11 last 73.46)
 - open/close 9741 (time 1788836213.93 total 140.13 last 69.20)
total: 10000 open/close in 144.01 seconds: 69.44 ops/second
 - unlinked 0 (time 1788836223 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.139@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.139@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.139@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg139-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-OST0000
oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg139-client.virtnet:  -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg139-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg139-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server
stop mds service on oleg139-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server
unloading modules via unload_modules_local on: 'oleg139-server'
oleg139-server: oleg139-server.virtnet: executing unload_modules_local
oleg139-server: kmemleak disabled
modules unloaded.
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2
pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2