== conf-sanity test 69: replace an OST with the same index ========================================================== 08:46:21 (1786797981)
start mds service on oleg325-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg325-server'
oleg325-server: oleg325-server.virtnet: executing load_modules_local
oleg325-server: Loading modules from /home/green/git/lustre-release/lustre
oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg325-server: detected 4 online CPUs by sysfs
oleg325-server: MODOPTS_LIBCFS=
oleg325-server: Force libcfs to create 2 CPU partitions
oleg325-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg325-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg325-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg325-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg325-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1
Started lustre-OST0000
oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg325-client.virtnet:  -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 961 (time 1786798075.82 total 10.00 last 96.05)
 - open/close 1831 (time 1786798085.83 total 20.02 last 86.89)
 - open/close 2529 (time 1786798095.84 total 30.03 last 69.73)
 - open/close 3247 (time 1786798105.86 total 40.05 last 71.64)
 - open/close 4162 (time 1786798115.87 total 50.06 last 91.43)
 - open/close 5094 (time 1786798125.88 total 60.06 last 93.14)
 - open/close 5955 (time 1786798135.88 total 70.07 last 86.06)
 - open/close 6862 (time 1786798145.88 total 80.07 last 90.69)
 - open/close 7632 (time 1786798155.89 total 90.08 last 76.92)
 - open/close 8434 (time 1786798165.89 total 100.08 last 80.18)
 - open/close 9166 (time 1786798175.91 total 110.10 last 73.07)
total: 10000 open/close in 118.91 seconds: 84.09 ops/second
 - unlinked 0 (time 1786798188 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 940 (time 1786798274.74 total 10.01 last 93.91)
 - open/close 1929 (time 1786798284.74 total 20.01 last 98.89)
 - open/close 2923 (time 1786798294.74 total 30.01 last 99.37)
 - open/close 3858 (time 1786798304.76 total 40.03 last 93.36)
 - open/close 4941 (time 1786798314.76 total 50.03 last 108.26)
 - open/close 5738 (time 1786798324.78 total 60.05 last 79.52)
 - open/close 6748 (time 1786798334.78 total 70.05 last 100.99)
 - open/close 7755 (time 1786798344.79 total 80.06 last 100.64)
 - open/close 8601 (time 1786798354.79 total 90.06 last 84.57)
 - open/close 9582 (time 1786798364.80 total 100.07 last 98.03)
total: 10000 open/close in 103.72 seconds: 96.41 ops/second
 - unlinked 0 (time 1786798372 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 840 (time 1786798456.96 total 10.01 last 83.95)
 - open/close 1807 (time 1786798466.96 total 20.01 last 96.65)
 - open/close 2760 (time 1786798476.98 total 30.02 last 95.16)
 - open/close 3735 (time 1786798486.98 total 40.03 last 97.46)
 - open/close 4748 (time 1786798496.99 total 50.03 last 101.26)
 - open/close 5771 (time 1786798506.99 total 60.04 last 102.27)
 - open/close 6701 (time 1786798517.00 total 70.05 last 92.91)
 - open/close 7285 (time 1786798527.01 total 80.06 last 58.31)
 - open/close 8065 (time 1786798537.03 total 90.07 last 77.90)
 - open/close 8788 (time 1786798547.03 total 100.08 last 72.26)
 - open/close 9686 (time 1786798557.03 total 110.08 last 89.78)
total: 10000 open/close in 112.98 seconds: 88.51 ops/second
 - unlinked 0 (time 1786798564 ; total 0 ; last 0)
total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second
 - open/close 848 (time 1786798643.80 total 10.01 last 84.73)
 - open/close 1555 (time 1786798653.80 total 20.01 last 70.66)
 - open/close 2614 (time 1786798663.80 total 30.01 last 105.89)
 - open/close 3558 (time 1786798673.81 total 40.02 last 94.37)
 - open/close 4457 (time 1786798683.81 total 50.02 last 89.88)
 - open/close 5571 (time 1786798693.82 total 60.03 last 111.24)
 - open/close 6361 (time 1786798703.83 total 70.04 last 78.96)
 - open/close 7141 (time 1786798713.85 total 80.07 last 77.80)
 - open/close 7961 (time 1786798723.87 total 90.08 last 81.90)
 - open/close 8570 (time 1786798733.87 total 100.08 last 60.86)
 - open/close 9500 (time 1786798743.88 total 110.09 last 92.96)
total: 10000 open/close in 115.30 seconds: 86.73 ops/second
 - unlinked 0 (time 1786798752 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 904 (time 1786798849.50 total 10.01 last 90.33)
 - open/close 1816 (time 1786798859.51 total 20.02 last 91.08)
 - open/close 2722 (time 1786798869.52 total 30.03 last 90.56)
 - open/close 3531 (time 1786798879.52 total 40.03 last 80.86)
 - open/close 4237 (time 1786798889.53 total 50.04 last 70.55)
 - open/close 4834 (time 1786798899.53 total 60.04 last 59.70)
 - open/close 5278 (time 1786798909.53 total 70.04 last 44.39)
 - open/close 6116 (time 1786798919.56 total 80.06 last 83.62)
 - open/close 7138 (time 1786798929.57 total 90.07 last 102.11)
 - open/close 8125 (time 1786798939.57 total 100.07 last 98.69)
 - open/close 9059 (time 1786798949.58 total 110.08 last 93.30)
total: 10000 open/close in 119.14 seconds: 83.94 ops/second
 - unlinked 0 (time 1786798962 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 893 (time 1786799044.80 total 10.00 last 89.26)
 - open/close 1828 (time 1786799054.80 total 20.01 last 93.47)
 - open/close 2665 (time 1786799064.80 total 30.01 last 83.69)
 - open/close 3440 (time 1786799074.82 total 40.03 last 77.36)
 - open/close 4353 (time 1786799084.84 total 50.04 last 91.15)
 - open/close 5241 (time 1786799094.85 total 60.05 last 88.73)
 - open/close 6188 (time 1786799104.85 total 70.06 last 94.65)
 - open/close 7180 (time 1786799114.85 total 80.06 last 99.20)
 - open/close 8099 (time 1786799124.87 total 90.08 last 91.73)
 - open/close 9064 (time 1786799134.87 total 100.08 last 96.46)
total: 10000 open/close in 109.58 seconds: 91.26 ops/second
 - unlinked 0 (time 1786799148 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 923 (time 1786799229.42 total 10.00 last 92.30)
 - open/close 1970 (time 1786799239.42 total 20.00 last 104.66)
 - open/close 2937 (time 1786799249.42 total 30.00 last 96.69)
 - open/close 3922 (time 1786799259.42 total 40.01 last 98.48)
 - open/close 4768 (time 1786799269.43 total 50.01 last 84.57)
 - open/close 5588 (time 1786799279.43 total 60.02 last 81.95)
 - open/close 6575 (time 1786799289.46 total 70.04 last 98.49)
 - open/close 7467 (time 1786799299.47 total 80.05 last 89.10)
 - open/close 8311 (time 1786799309.47 total 90.05 last 84.40)
 - open/close 9291 (time 1786799319.47 total 100.05 last 97.96)
total: 10000 open/close in 107.12 seconds: 93.36 ops/second
 - unlinked 0 (time 1786799330 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 944 (time 1786799419.42 total 10.01 last 94.33)
 - open/close 1938 (time 1786799429.43 total 20.02 last 99.27)
 - open/close 2546 (time 1786799439.44 total 30.03 last 60.75)
 - open/close 3055 (time 1786799449.45 total 40.04 last 50.83)
 - open/close 3728 (time 1786799459.47 total 50.06 last 67.17)
 - open/close 4251 (time 1786799469.48 total 60.07 last 52.27)
 - open/close 5210 (time 1786799479.48 total 70.07 last 95.85)
 - open/close 6044 (time 1786799489.49 total 80.08 last 83.31)
 - open/close 6906 (time 1786799499.50 total 90.09 last 86.14)
 - open/close 7885 (time 1786799509.50 total 100.10 last 97.87)
 - open/close 8825 (time 1786799519.51 total 110.10 last 93.97)
 - open/close 9746 (time 1786799529.51 total 120.11 last 92.03)
total: 10000 open/close in 122.61 seconds: 81.56 ops/second
 - unlinked 0 (time 1786799535 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 787 (time 1786799623.96 total 10.00 last 78.67)
 - open/close 1765 (time 1786799633.97 total 20.01 last 97.77)
 - open/close 2743 (time 1786799643.97 total 30.01 last 97.76)
 - open/close 3561 (time 1786799653.98 total 40.02 last 81.76)
 - open/close 4342 (time 1786799663.98 total 50.02 last 78.06)
 - open/close 5236 (time 1786799673.99 total 60.03 last 89.33)
 - open/close 6028 (time 1786799683.99 total 70.03 last 79.19)
 - open/close 6878 (time 1786799693.99 total 80.03 last 84.98)
 - open/close 7845 (time 1786799704.00 total 90.04 last 96.68)
 - open/close 8775 (time 1786799714.00 total 100.04 last 92.95)
 - open/close 9695 (time 1786799724.01 total 110.05 last 91.93)
total: 10000 open/close in 114.17 seconds: 87.59 ops/second
 - unlinked 0 (time 1786799733 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 729 (time 1786799823.47 total 10.00 last 72.88)
 - open/close 1665 (time 1786799833.48 total 20.01 last 93.57)
 - open/close 2610 (time 1786799843.48 total 30.01 last 94.48)
 - open/close 3651 (time 1786799853.48 total 40.01 last 104.09)
 - open/close 4488 (time 1786799863.48 total 50.01 last 83.68)
 - open/close 5237 (time 1786799873.49 total 60.02 last 74.83)
 - open/close 6070 (time 1786799883.51 total 70.04 last 83.12)
 - open/close 6951 (time 1786799893.53 total 80.05 last 87.99)
 - open/close 8125 (time 1786799903.53 total 90.06 last 117.33)
 - open/close 9058 (time 1786799913.54 total 100.07 last 93.25)
total: 10000 open/close in 108.79 seconds: 91.92 ops/second
 - unlinked 0 (time 1786799926 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg325-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg325-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg325-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1
Started lustre-OST0000
oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg325-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg325-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg325-client.virtnet:  -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre
On OST0, 20520 used inodes rc=0
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg325-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg325-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.125@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.125@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg325-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1
Started lustre-OST0000
oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg325-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg325-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg325-client.virtnet:  -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg325-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg325-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server
stop mds service on oleg325-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server
stop mds service on oleg325-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server

unloading modules via unload_modules_local on: 'oleg325-server'
oleg325-server: oleg325-server.virtnet: executing unload_modules_local
oleg325-server: 
modules unloaded.
pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2
pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2
pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2