== conf-sanity test 69: replace an OST with the same index ========================================================== 22:24:34 (1788575074)
start mds service on oleg411-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1944872 Sep  4 04:21 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg411-server'
oleg411-server: oleg411-server.virtnet: executing load_modules_local
oleg411-server: kmemleak disabled
oleg411-server: Loading modules from /home/green/git/lustre-release/lustre
oleg411-server: detected 4 online CPUs by sysfs
oleg411-server: MODOPTS_LIBCFS=
oleg411-server: Force libcfs to create 2 CPU partitions
oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 553 (time 1788575170.51 total 10.00 last 55.28)
 - open/close 1140 (time 1788575180.51 total 20.01 last 58.68)
 - open/close 1782 (time 1788575190.52 total 30.01 last 64.17)
 - open/close 2402 (time 1788575200.53 total 40.02 last 61.95)
 - open/close 2972 (time 1788575210.54 total 50.03 last 56.93)
 - open/close 3541 (time 1788575220.55 total 60.04 last 56.84)
 - open/close 4140 (time 1788575230.55 total 70.05 last 59.88)
 - open/close 4735 (time 1788575240.56 total 80.06 last 59.44)
 - open/close 5343 (time 1788575250.59 total 90.08 last 60.64)
 - open/close 5966 (time 1788575260.60 total 100.09 last 62.26)
 - open/close 6616 (time 1788575270.60 total 110.10 last 64.94)
 - open/close 7180 (time 1788575280.61 total 120.11 last 56.35)
 - open/close 7826 (time 1788575290.63 total 130.12 last 64.50)
 - open/close 8520 (time 1788575300.64 total 140.13 last 69.32)
 - open/close 9162 (time 1788575310.64 total 150.14 last 64.18)
 - open/close 9748 (time 1788575320.65 total 160.14 last 58.58)
total: 10000 open/close in 164.20 seconds: 60.90 ops/second
 - unlinked 0 (time 1788575329 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 584 (time 1788575451.96 total 10.02 last 58.30)
 - open/close 1246 (time 1788575461.96 total 20.02 last 66.17)
 - open/close 1868 (time 1788575471.98 total 30.03 last 62.13)
 - open/close 2557 (time 1788575481.99 total 40.05 last 68.80)
 - open/close 3172 (time 1788575491.99 total 50.05 last 61.50)
 - open/close 3828 (time 1788575501.99 total 60.05 last 65.58)
 - open/close 4482 (time 1788575512.00 total 70.05 last 65.39)
 - open/close 5159 (time 1788575522.00 total 80.06 last 67.68)
 - open/close 5768 (time 1788575532.02 total 90.07 last 60.79)
 - open/close 6387 (time 1788575542.03 total 100.09 last 61.83)
 - open/close 6982 (time 1788575552.06 total 110.12 last 59.32)
 - open/close 7634 (time 1788575562.07 total 120.13 last 65.14)
 - open/close 8185 (time 1788575572.07 total 130.13 last 55.10)
 - open/close 8794 (time 1788575582.08 total 140.13 last 60.86)
 - open/close 9375 (time 1788575592.08 total 150.13 last 58.10)
 - open/close 9962 (time 1788575602.09 total 160.15 last 58.59)
total: 10000 open/close in 160.73 seconds: 62.22 ops/second
 - unlinked 0 (time 1788575607 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 689 (time 1788575716.86 total 10.00 last 68.89)
 - open/close 1337 (time 1788575726.87 total 20.01 last 64.73)
 - open/close 1963 (time 1788575736.89 total 30.03 last 62.48)
 - open/close 2529 (time 1788575746.90 total 40.04 last 56.54)
 - open/close 3048 (time 1788575756.90 total 50.04 last 51.89)
 - open/close 3728 (time 1788575766.91 total 60.05 last 67.94)
 - open/close 4374 (time 1788575776.91 total 70.06 last 64.57)
 - open/close 5054 (time 1788575786.93 total 80.07 last 67.93)
 - open/close 5703 (time 1788575796.93 total 90.08 last 64.84)
 - open/close 6346 (time 1788575806.94 total 100.08 last 64.27)
 - open/close 6998 (time 1788575816.94 total 110.09 last 65.16)
 - open/close 7550 (time 1788575826.98 total 120.13 last 54.99)
 - open/close 8033 (time 1788575837.01 total 130.15 last 48.17)
 - open/close 8640 (time 1788575847.01 total 140.16 last 60.69)
 - open/close 9258 (time 1788575857.01 total 150.16 last 61.80)
 - open/close 9883 (time 1788575867.02 total 160.17 last 62.44)
total: 10000 open/close in 161.87 seconds: 61.78 ops/second
 - unlinked 0 (time 1788575873 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 668 (time 1788575992.44 total 10.02 last 66.70)
 - open/close 1356 (time 1788576002.45 total 20.02 last 68.77)
 - open/close 1978 (time 1788576012.45 total 30.03 last 62.17)
 - open/close 2524 (time 1788576022.49 total 40.06 last 54.39)
 - open/close 3042 (time 1788576032.50 total 50.07 last 51.79)
 - open/close 3647 (time 1788576042.50 total 60.07 last 60.50)
 - open/close 4270 (time 1788576052.51 total 70.08 last 62.25)
 - open/close 4905 (time 1788576062.52 total 80.09 last 63.38)
 - open/close 5480 (time 1788576072.53 total 90.10 last 57.45)
 - open/close 6199 (time 1788576082.54 total 100.11 last 71.82)
 - open/close 6838 (time 1788576092.55 total 110.12 last 63.88)
 - open/close 7475 (time 1788576102.55 total 120.12 last 63.69)
 - open/close 8092 (time 1788576112.55 total 130.12 last 61.69)
 - open/close 8714 (time 1788576122.56 total 140.13 last 62.14)
 - open/close 9318 (time 1788576132.56 total 150.13 last 60.40)
 - open/close 9978 (time 1788576142.56 total 160.13 last 65.97)
total: 10000 open/close in 160.45 seconds: 62.32 ops/second
 - unlinked 0 (time 1788576147 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 732 (time 1788576251.64 total 10.04 last 72.92)
 - open/close 1364 (time 1788576261.66 total 20.06 last 63.09)
 - open/close 1969 (time 1788576271.66 total 30.06 last 60.47)
 - open/close 2577 (time 1788576281.67 total 40.07 last 60.76)
 - open/close 3185 (time 1788576291.68 total 50.08 last 60.72)
 - open/close 3887 (time 1788576301.69 total 60.09 last 70.14)
 - open/close 4465 (time 1788576311.72 total 70.11 last 57.65)
 - open/close 4926 (time 1788576321.73 total 80.13 last 46.05)
 - open/close 5526 (time 1788576331.73 total 90.13 last 59.97)
 - open/close 6144 (time 1788576341.73 total 100.13 last 61.79)
 - open/close 6799 (time 1788576351.74 total 110.14 last 65.47)
 - open/close 7457 (time 1788576361.75 total 120.15 last 65.71)
 - open/close 8117 (time 1788576371.75 total 130.15 last 65.99)
 - open/close 8791 (time 1788576381.76 total 140.15 last 67.38)
 - open/close 9464 (time 1788576391.77 total 150.16 last 67.24)
total: 10000 open/close in 158.11 seconds: 63.25 ops/second
 - unlinked 0 (time 1788576403 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 567 (time 1788576518.53 total 10.03 last 56.56)
 - open/close 1208 (time 1788576528.55 total 20.05 last 63.95)
 - open/close 1820 (time 1788576538.56 total 30.06 last 61.15)
 - open/close 2413 (time 1788576548.57 total 40.07 last 59.22)
 - open/close 2971 (time 1788576558.58 total 50.08 last 55.74)
 - open/close 3463 (time 1788576568.63 total 60.13 last 48.98)
 - open/close 4009 (time 1788576578.64 total 70.13 last 54.56)
 - open/close 4696 (time 1788576588.66 total 80.16 last 68.55)
 - open/close 5349 (time 1788576598.67 total 90.17 last 65.21)
 - open/close 6034 (time 1788576608.69 total 100.18 last 68.41)
 - open/close 6648 (time 1788576618.69 total 110.19 last 61.38)
 - open/close 7223 (time 1788576628.69 total 120.19 last 57.48)
 - open/close 7829 (time 1788576638.70 total 130.20 last 60.53)
 - open/close 8519 (time 1788576648.71 total 140.21 last 68.95)
 - open/close 9192 (time 1788576658.72 total 150.22 last 67.21)
 - open/close 9796 (time 1788576668.73 total 160.23 last 60.34)
total: 10000 open/close in 163.64 seconds: 61.11 ops/second
 - unlinked 0 (time 1788576677 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 658 (time 1788576790.08 total 10.02 last 65.66)
 - open/close 1298 (time 1788576800.10 total 20.04 last 63.87)
 - open/close 1901 (time 1788576810.12 total 30.06 last 60.21)
 - open/close 2560 (time 1788576820.13 total 40.07 last 65.81)
 - open/close 3171 (time 1788576830.14 total 50.08 last 61.03)
 - open/close 3797 (time 1788576840.15 total 60.09 last 62.55)
 - open/close 4448 (time 1788576850.16 total 70.10 last 65.03)
 - open/close 5090 (time 1788576860.16 total 80.10 last 64.18)
 - open/close 5726 (time 1788576870.18 total 90.11 last 63.53)
 - open/close 6329 (time 1788576880.19 total 100.12 last 60.23)
 - open/close 6911 (time 1788576890.22 total 110.15 last 58.03)
 - open/close 7532 (time 1788576900.23 total 120.16 last 62.03)
 - open/close 8155 (time 1788576910.25 total 130.18 last 62.18)
 - open/close 8760 (time 1788576920.25 total 140.19 last 60.48)
 - open/close 9377 (time 1788576930.25 total 150.19 last 61.70)
total: 10000 open/close in 159.67 seconds: 62.63 ops/second
 - unlinked 0 (time 1788576943 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 572 (time 1788577055.96 total 10.01 last 57.16)
 - open/close 1201 (time 1788577065.97 total 20.01 last 62.88)
 - open/close 1791 (time 1788577075.98 total 30.03 last 58.91)
 - open/close 2469 (time 1788577085.99 total 40.03 last 67.75)
 - open/close 3134 (time 1788577096.00 total 50.04 last 66.45)
 - open/close 3730 (time 1788577106.01 total 60.06 last 59.50)
 - open/close 4359 (time 1788577116.01 total 70.06 last 62.89)
 - open/close 4919 (time 1788577126.03 total 80.08 last 55.91)
 - open/close 5556 (time 1788577136.05 total 90.10 last 63.58)
 - open/close 6147 (time 1788577146.06 total 100.10 last 59.05)
 - open/close 6757 (time 1788577156.06 total 110.11 last 60.97)
 - open/close 7377 (time 1788577166.07 total 120.12 last 61.93)
 - open/close 8023 (time 1788577176.08 total 130.13 last 64.56)
 - open/close 8691 (time 1788577186.08 total 140.13 last 66.78)
 - open/close 9369 (time 1788577196.09 total 150.14 last 67.75)
total: 10000 open/close in 159.79 seconds: 62.58 ops/second
 - unlinked 0 (time 1788577209 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 565 (time 1788577328.81 total 10.00 last 56.48)
 - open/close 1156 (time 1788577338.82 total 20.02 last 59.03)
 - open/close 1726 (time 1788577348.83 total 30.03 last 56.94)
 - open/close 2337 (time 1788577358.84 total 40.03 last 61.07)
 - open/close 3015 (time 1788577368.84 total 50.03 last 67.80)
 - open/close 3609 (time 1788577378.86 total 60.05 last 59.31)
 - open/close 4242 (time 1788577388.86 total 70.05 last 63.28)
 - open/close 4898 (time 1788577398.87 total 80.06 last 65.53)
 - open/close 5545 (time 1788577408.88 total 90.07 last 64.62)
 - open/close 6186 (time 1788577418.89 total 100.08 last 64.07)
 - open/close 6841 (time 1788577428.91 total 110.10 last 65.37)
 - open/close 7478 (time 1788577438.91 total 120.10 last 63.70)
 - open/close 8100 (time 1788577448.91 total 130.11 last 62.16)
 - open/close 8706 (time 1788577458.93 total 140.12 last 60.53)
 - open/close 9334 (time 1788577468.96 total 150.15 last 62.60)
 - open/close 9947 (time 1788577478.96 total 160.15 last 61.29)
total: 10000 open/close in 161.06 seconds: 62.09 ops/second
 - unlinked 0 (time 1788577484 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 622 (time 1788577599.07 total 10.01 last 62.15)
 - open/close 1245 (time 1788577609.09 total 20.03 last 62.17)
 - open/close 1809 (time 1788577619.10 total 30.04 last 56.35)
 - open/close 2393 (time 1788577629.11 total 40.04 last 58.36)
 - open/close 3025 (time 1788577639.14 total 50.07 last 63.04)
 - open/close 3649 (time 1788577649.15 total 60.08 last 62.32)
 - open/close 4271 (time 1788577659.16 total 70.09 last 62.15)
 - open/close 4897 (time 1788577669.16 total 80.10 last 62.56)
 - open/close 5524 (time 1788577679.16 total 90.10 last 62.69)
 - open/close 6097 (time 1788577689.18 total 100.11 last 57.22)
 - open/close 6698 (time 1788577699.18 total 110.11 last 60.09)
 - open/close 7352 (time 1788577709.18 total 120.12 last 65.38)
 - open/close 7974 (time 1788577719.18 total 130.12 last 62.18)
 - open/close 8597 (time 1788577729.19 total 140.12 last 62.29)
 - open/close 9252 (time 1788577739.19 total 150.12 last 65.49)
 - open/close 9878 (time 1788577749.19 total 160.13 last 62.56)
total: 10000 open/close in 162.08 seconds: 61.70 ops/second
 - unlinked 0 (time 1788577755 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.111@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.111@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.111@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg411-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1
Started lustre-OST0000
oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg411-client.virtnet:  -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg411-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg411-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server
stop mds service on oleg411-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server
unloading modules via unload_modules_local on: 'oleg411-server'
oleg411-server: oleg411-server.virtnet: executing unload_modules_local
oleg411-server: kmemleak disabled
modules unloaded.
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2
pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2