== conf-sanity test 69: replace an OST with the same index ========================================================== 04:43:29 (1787733809)
start mds service on oleg335-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg335-server'
oleg335-server: oleg335-server.virtnet: executing load_modules_local
oleg335-server: Loading modules from /home/green/git/lustre-release/lustre
oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg335-server: detected 4 online CPUs by sysfs
oleg335-server: MODOPTS_LIBCFS=
oleg335-server: Force libcfs to create 2 CPU partitions
oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg335-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-OST0000
oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg335-client.virtnet:  -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 677 (time 1787733896.95 total 10.03 last 67.52)
 - open/close 1327 (time 1787733906.96 total 20.04 last 64.94)
 - open/close 2000 (time 1787733916.97 total 30.05 last 67.20)
 - open/close 2688 (time 1787733926.98 total 40.05 last 68.77)
 - open/close 3359 (time 1787733936.99 total 50.07 last 66.98)
 - open/close 4033 (time 1787733946.99 total 60.07 last 67.40)
 - open/close 4681 (time 1787733957.01 total 70.09 last 64.73)
 - open/close 5384 (time 1787733967.01 total 80.09 last 70.24)
 - open/close 6048 (time 1787733977.02 total 90.10 last 66.33)
 - open/close 6702 (time 1787733987.04 total 100.11 last 65.33)
 - open/close 7401 (time 1787733997.04 total 110.12 last 69.88)
 - open/close 8078 (time 1787734007.04 total 120.12 last 67.66)
 - open/close 8725 (time 1787734017.05 total 130.12 last 64.68)
 - open/close 9322 (time 1787734027.05 total 140.13 last 59.68)
 - open/close 9900 (time 1787734037.05 total 150.13 last 57.79)
total: 10000 open/close in 151.77 seconds: 65.89 ops/second
 - unlinked 0 (time 1787734043 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 701 (time 1787734146.44 total 10.01 last 70.01)
 - open/close 1346 (time 1787734156.45 total 20.03 last 64.41)
 - open/close 2093 (time 1787734166.46 total 30.04 last 74.64)
 - open/close 2766 (time 1787734176.47 total 40.04 last 67.23)
 - open/close 3493 (time 1787734186.48 total 50.06 last 72.62)
 - open/close 4228 (time 1787734196.49 total 60.06 last 73.44)
 - open/close 4981 (time 1787734206.50 total 70.07 last 75.24)
 - open/close 5746 (time 1787734216.50 total 80.08 last 76.45)
 - open/close 6384 (time 1787734226.51 total 90.08 last 63.77)
 - open/close 7057 (time 1787734236.52 total 100.10 last 67.20)
 - open/close 7668 (time 1787734246.53 total 110.10 last 61.07)
 - open/close 8175 (time 1787734256.53 total 120.11 last 50.68)
 - open/close 8746 (time 1787734266.54 total 130.11 last 57.09)
 - open/close 9342 (time 1787734276.54 total 140.11 last 59.58)
total: 10000 open/close in 149.71 seconds: 66.79 ops/second
 - unlinked 0 (time 1787734289 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 662 (time 1787734385.25 total 10.01 last 66.16)
 - open/close 1339 (time 1787734395.27 total 20.02 last 67.60)
 - open/close 2037 (time 1787734405.27 total 30.02 last 69.80)
 - open/close 2716 (time 1787734415.27 total 40.02 last 67.90)
 - open/close 3444 (time 1787734425.29 total 50.04 last 72.66)
 - open/close 4137 (time 1787734435.29 total 60.05 last 69.26)
 - open/close 4843 (time 1787734445.31 total 70.06 last 70.52)
 - open/close 5497 (time 1787734455.31 total 80.07 last 65.34)
 - open/close 6126 (time 1787734465.33 total 90.08 last 62.84)
 - open/close 6772 (time 1787734475.33 total 100.08 last 64.60)
 - open/close 7521 (time 1787734485.34 total 110.09 last 74.82)
 - open/close 8273 (time 1787734495.34 total 120.09 last 75.16)
 - open/close 9003 (time 1787734505.35 total 130.10 last 72.93)
 - open/close 9682 (time 1787734515.36 total 140.11 last 67.86)
total: 10000 open/close in 143.85 seconds: 69.52 ops/second
 - unlinked 0 (time 1787734522 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 762 (time 1787734616.89 total 10.01 last 76.16)
 - open/close 1484 (time 1787734626.89 total 20.01 last 72.19)
 - open/close 2185 (time 1787734636.90 total 30.01 last 70.06)
 - open/close 2861 (time 1787734646.90 total 40.01 last 67.59)
 - open/close 3622 (time 1787734656.91 total 50.02 last 76.04)
 - open/close 4305 (time 1787734666.92 total 60.04 last 68.19)
 - open/close 4991 (time 1787734676.95 total 70.06 last 68.45)
 - open/close 5685 (time 1787734686.96 total 80.07 last 69.33)
 - open/close 6377 (time 1787734696.96 total 90.07 last 69.19)
 - open/close 7077 (time 1787734706.96 total 100.08 last 69.95)
 - open/close 7783 (time 1787734716.97 total 110.08 last 70.57)
 - open/close 8479 (time 1787734726.97 total 120.09 last 69.56)
 - open/close 9196 (time 1787734736.98 total 130.10 last 71.64)
 - open/close 9834 (time 1787734746.99 total 140.10 last 63.78)
total: 10000 open/close in 142.11 seconds: 70.37 ops/second
 - unlinked 0 (time 1787734754 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 649 (time 1787734866.95 total 10.01 last 64.85)
 - open/close 1291 (time 1787734876.96 total 20.01 last 64.16)
 - open/close 1943 (time 1787734886.97 total 30.03 last 65.11)
 - open/close 2619 (time 1787734896.98 total 40.04 last 67.51)
 - open/close 3299 (time 1787734906.99 total 50.05 last 67.93)
 - open/close 3985 (time 1787734917.00 total 60.06 last 68.55)
 - open/close 4602 (time 1787734927.01 total 70.06 last 61.67)
 - open/close 5241 (time 1787734937.01 total 80.07 last 63.88)
 - open/close 5876 (time 1787734947.02 total 90.08 last 63.40)
 - open/close 6534 (time 1787734957.03 total 100.09 last 65.77)
 - open/close 7237 (time 1787734967.03 total 110.09 last 70.28)
 - open/close 7916 (time 1787734977.05 total 120.11 last 67.79)
 - open/close 8627 (time 1787734987.06 total 130.11 last 71.05)
 - open/close 9354 (time 1787734997.06 total 140.11 last 72.69)
total: 10000 open/close in 150.09 seconds: 66.63 ops/second
 - unlinked 0 (time 1787735011 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 724 (time 1787735122.94 total 10.01 last 72.34)
 - open/close 1363 (time 1787735132.94 total 20.02 last 63.85)
 - open/close 2034 (time 1787735142.95 total 30.02 last 67.05)
 - open/close 2721 (time 1787735152.96 total 40.04 last 68.62)
 - open/close 3373 (time 1787735162.97 total 50.04 last 65.15)
 - open/close 3971 (time 1787735172.98 total 60.05 last 59.76)
 - open/close 4687 (time 1787735182.99 total 70.06 last 71.52)
 - open/close 5335 (time 1787735192.99 total 80.07 last 64.76)
 - open/close 6014 (time 1787735203.00 total 90.07 last 67.89)
 - open/close 6718 (time 1787735213.01 total 100.08 last 70.29)
 - open/close 7441 (time 1787735223.02 total 110.09 last 72.23)
 - open/close 8152 (time 1787735233.02 total 120.10 last 71.08)
 - open/close 8886 (time 1787735243.02 total 130.10 last 73.40)
 - open/close 9595 (time 1787735253.04 total 140.11 last 70.81)
total: 10000 open/close in 146.06 seconds: 68.46 ops/second
 - unlinked 0 (time 1787735264 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 700 (time 1787735365.70 total 10.02 last 69.88)
 - open/close 1371 (time 1787735375.70 total 20.02 last 67.06)
 - open/close 2014 (time 1787735385.72 total 30.04 last 64.22)
 - open/close 2711 (time 1787735395.72 total 40.05 last 69.64)
 - open/close 3365 (time 1787735405.72 total 50.05 last 65.40)
 - open/close 4019 (time 1787735415.73 total 60.05 last 65.38)
 - open/close 4677 (time 1787735425.75 total 70.07 last 65.66)
 - open/close 5324 (time 1787735435.75 total 80.07 last 64.68)
 - open/close 5982 (time 1787735445.76 total 90.08 last 65.73)
 - open/close 6675 (time 1787735455.77 total 100.09 last 69.24)
 - open/close 7381 (time 1787735465.78 total 110.10 last 70.56)
 - open/close 8152 (time 1787735475.78 total 120.10 last 77.06)
 - open/close 8888 (time 1787735485.80 total 130.12 last 73.44)
 - open/close 9585 (time 1787735495.82 total 140.14 last 69.58)
total: 10000 open/close in 146.54 seconds: 68.24 ops/second
 - unlinked 0 (time 1787735506 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 756 (time 1787735603.19 total 10.01 last 75.56)
 - open/close 1492 (time 1787735613.20 total 20.02 last 73.52)
 - open/close 2235 (time 1787735623.21 total 30.03 last 74.22)
 - open/close 2979 (time 1787735633.21 total 40.03 last 74.37)
 - open/close 3742 (time 1787735643.22 total 50.04 last 76.22)
 - open/close 4544 (time 1787735653.23 total 60.05 last 80.12)
 - open/close 5302 (time 1787735663.24 total 70.06 last 75.74)
 - open/close 6091 (time 1787735673.24 total 80.06 last 78.85)
 - open/close 6823 (time 1787735683.25 total 90.07 last 73.15)
 - open/close 7598 (time 1787735693.25 total 100.07 last 77.49)
 - open/close 8369 (time 1787735703.26 total 110.08 last 77.04)
 - open/close 9124 (time 1787735713.26 total 120.08 last 75.49)
 - open/close 9945 (time 1787735723.27 total 130.09 last 82.06)
total: 10000 open/close in 130.70 seconds: 76.51 ops/second
 - unlinked 0 (time 1787735727 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 580 (time 1787735824.71 total 10.02 last 57.90)
 - open/close 1330 (time 1787735834.72 total 20.03 last 74.91)
 - open/close 1993 (time 1787735844.73 total 30.04 last 66.25)
 - open/close 2695 (time 1787735854.75 total 40.05 last 70.08)
 - open/close 3475 (time 1787735864.75 total 50.06 last 77.99)
 - open/close 4201 (time 1787735874.76 total 60.07 last 72.49)
 - open/close 5071 (time 1787735884.77 total 70.08 last 86.96)
 - open/close 5746 (time 1787735894.78 total 80.08 last 67.44)
 - open/close 6362 (time 1787735904.78 total 90.09 last 61.59)
 - open/close 7072 (time 1787735914.78 total 100.09 last 70.97)
 - open/close 7827 (time 1787735924.79 total 110.10 last 75.41)
 - open/close 8541 (time 1787735934.80 total 120.11 last 71.35)
 - open/close 9280 (time 1787735944.81 total 130.12 last 73.85)
 - open/close 9911 (time 1787735954.82 total 140.13 last 63.05)
total: 10000 open/close in 141.35 seconds: 70.75 ops/second
 - unlinked 0 (time 1787735960 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 703 (time 1787736055.74 total 10.01 last 70.22)
 - open/close 1431 (time 1787736065.74 total 20.01 last 72.80)
 - open/close 2170 (time 1787736075.74 total 30.02 last 73.84)
 - open/close 2917 (time 1787736085.75 total 40.03 last 74.64)
 - open/close 3677 (time 1787736095.76 total 50.04 last 75.90)
 - open/close 4386 (time 1787736105.77 total 60.05 last 70.86)
 - open/close 5073 (time 1787736115.77 total 70.05 last 68.69)
 - open/close 5750 (time 1787736125.79 total 80.06 last 67.61)
 - open/close 6462 (time 1787736135.79 total 90.06 last 71.19)
 - open/close 7129 (time 1787736145.79 total 100.07 last 66.65)
 - open/close 7837 (time 1787736155.79 total 110.07 last 70.80)
 - open/close 8575 (time 1787736165.80 total 120.07 last 73.79)
 - open/close 9280 (time 1787736175.81 total 130.08 last 70.41)
total: 10000 open/close in 139.75 seconds: 71.56 ops/second
 - unlinked 0 (time 1787736188 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg335-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg335-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg335-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-OST0000
oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg335-client.virtnet:  -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg335-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg335-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.135@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.135@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.135@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg335-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-OST0000
oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg335-client.virtnet:  -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg335-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg335-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server
stop mds service on oleg335-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server

unloading modules via unload_modules_local on: 'oleg335-server'
oleg335-server: oleg335-server.virtnet: executing unload_modules_local
oleg335-server: 
modules unloaded.
pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2
pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2