== conf-sanity test 69: replace an OST with the same index ========================================================== 03:14:56 (1789110896)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 692 (time 1789110990.41 total 10.01 last 69.14)
 - open/close 1335 (time 1789111000.41 total 20.01 last 64.28)
 - open/close 1980 (time 1789111010.42 total 30.02 last 64.45)
 - open/close 2710 (time 1789111020.42 total 40.02 last 72.99)
 - open/close 3570 (time 1789111030.43 total 50.03 last 85.92)
 - open/close 4364 (time 1789111040.44 total 60.04 last 79.35)
 - open/close 5131 (time 1789111050.45 total 70.05 last 76.56)
 - open/close 5719 (time 1789111060.46 total 80.06 last 58.79)
 - open/close 6551 (time 1789111070.46 total 90.06 last 83.15)
 - open/close 7289 (time 1789111080.48 total 100.09 last 73.64)
 - open/close 8071 (time 1789111090.49 total 110.09 last 78.15)
 - open/close 8826 (time 1789111100.49 total 120.09 last 75.48)
 - open/close 9678 (time 1789111110.50 total 130.10 last 85.16)
total: 10000 open/close in 134.01 seconds: 74.62 ops/second
 - unlinked 0 (time 1789111118 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 837 (time 1789111224.94 total 10.01 last 83.61)
 - open/close 1584 (time 1789111234.94 total 20.02 last 74.66)
 - open/close 2393 (time 1789111244.95 total 30.02 last 80.84)
 - open/close 3163 (time 1789111254.95 total 40.03 last 76.99)
 - open/close 3914 (time 1789111264.98 total 50.05 last 74.92)
 - open/close 4705 (time 1789111274.98 total 60.05 last 79.10)
 - open/close 5529 (time 1789111284.99 total 70.07 last 82.28)
 - open/close 6267 (time 1789111295.01 total 80.08 last 73.69)
 - open/close 6923 (time 1789111305.01 total 90.08 last 65.60)
 - open/close 7702 (time 1789111315.01 total 100.09 last 77.85)
 - open/close 8423 (time 1789111325.02 total 110.09 last 72.06)
 - open/close 9172 (time 1789111335.03 total 120.10 last 74.83)
 - open/close 9888 (time 1789111345.04 total 130.11 last 71.55)
total: 10000 open/close in 131.86 seconds: 75.84 ops/second
 - unlinked 0 (time 1789111352 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 802 (time 1789111453.89 total 10.01 last 80.16)
 - open/close 1604 (time 1789111463.89 total 20.01 last 80.17)
 - open/close 2391 (time 1789111473.90 total 30.02 last 78.65)
 - open/close 3069 (time 1789111483.90 total 40.02 last 67.78)
 - open/close 3817 (time 1789111493.92 total 50.04 last 74.61)
 - open/close 4498 (time 1789111503.93 total 60.05 last 68.08)
 - open/close 5291 (time 1789111513.93 total 70.05 last 79.28)
 - open/close 6090 (time 1789111523.93 total 80.05 last 79.87)
 - open/close 6853 (time 1789111533.94 total 90.06 last 76.28)
 - open/close 7591 (time 1789111543.94 total 100.06 last 73.79)
 - open/close 8370 (time 1789111553.95 total 110.07 last 77.77)
 - open/close 9135 (time 1789111563.96 total 120.08 last 76.46)
 - open/close 9924 (time 1789111573.96 total 130.08 last 78.89)
total: 10000 open/close in 131.25 seconds: 76.19 ops/second
 - unlinked 0 (time 1789111580 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 842 (time 1789111679.57 total 10.01 last 84.12)
 - open/close 1611 (time 1789111689.57 total 20.01 last 76.89)
 - open/close 2385 (time 1789111699.58 total 30.02 last 77.34)
 - open/close 3138 (time 1789111709.58 total 40.02 last 75.30)
 - open/close 3870 (time 1789111719.58 total 50.02 last 73.15)
 - open/close 4761 (time 1789111729.59 total 60.03 last 89.08)
 - open/close 5619 (time 1789111739.60 total 70.04 last 85.73)
 - open/close 6548 (time 1789111749.60 total 80.04 last 92.89)
 - open/close 7420 (time 1789111759.61 total 90.05 last 87.06)
 - open/close 8159 (time 1789111769.62 total 100.06 last 73.88)
 - open/close 8665 (time 1789111779.63 total 110.07 last 50.53)
 - open/close 9086 (time 1789111789.64 total 120.08 last 42.05)
 - open/close 9500 (time 1789111799.65 total 130.09 last 41.35)
total: 10000 open/close in 138.18 seconds: 72.37 ops/second
 - unlinked 0 (time 1789111813 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 872 (time 1789111907.23 total 10.01 last 87.14)
 - open/close 1698 (time 1789111917.24 total 20.01 last 82.53)
 - open/close 2398 (time 1789111927.24 total 30.02 last 69.99)
 - open/close 3330 (time 1789111937.24 total 40.02 last 93.16)
 - open/close 4232 (time 1789111947.25 total 50.03 last 90.16)
 - open/close 5115 (time 1789111957.28 total 60.05 last 88.06)
 - open/close 6051 (time 1789111967.28 total 70.06 last 93.51)
 - open/close 7000 (time 1789111977.29 total 80.06 last 94.88)
 - open/close 7801 (time 1789111987.30 total 90.07 last 80.03)
 - open/close 8589 (time 1789111997.30 total 100.07 last 78.80)
 - open/close 9335 (time 1789112007.30 total 110.08 last 74.53)
total: 10000 open/close in 118.30 seconds: 84.53 ops/second
 - unlinked 0 (time 1789112019 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 676 (time 1789112109.64 total 10.01 last 67.57)
 - open/close 1500 (time 1789112119.64 total 20.01 last 82.34)
 - open/close 2415 (time 1789112129.65 total 30.02 last 91.45)
 - open/close 3348 (time 1789112139.65 total 40.02 last 93.27)
 - open/close 4208 (time 1789112149.66 total 50.03 last 85.90)
 - open/close 5060 (time 1789112159.67 total 60.04 last 85.14)
 - open/close 5740 (time 1789112169.69 total 70.06 last 67.89)
 - open/close 6477 (time 1789112179.69 total 80.06 last 73.68)
 - open/close 7194 (time 1789112189.69 total 90.06 last 71.67)
 - open/close 8100 (time 1789112199.70 total 100.07 last 90.57)
 - open/close 8977 (time 1789112209.70 total 110.07 last 87.69)
 - open/close 9568 (time 1789112219.70 total 120.07 last 59.08)
total: 10000 open/close in 126.28 seconds: 79.19 ops/second
 - unlinked 0 (time 1789112230 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 784 (time 1789112314.48 total 10.02 last 78.25)
 - open/close 1672 (time 1789112324.49 total 20.03 last 88.70)
 - open/close 2431 (time 1789112334.52 total 30.06 last 75.68)
 - open/close 3099 (time 1789112344.52 total 40.06 last 66.79)
 - open/close 4005 (time 1789112354.53 total 50.06 last 90.59)
 - open/close 4851 (time 1789112364.53 total 60.06 last 84.59)
 - open/close 5722 (time 1789112374.53 total 70.07 last 87.05)
 - open/close 6563 (time 1789112384.53 total 80.07 last 84.10)
 - open/close 7384 (time 1789112394.54 total 90.07 last 82.07)
 - open/close 8275 (time 1789112404.54 total 100.08 last 89.05)
 - open/close 9029 (time 1789112414.55 total 110.09 last 75.32)
 - open/close 9745 (time 1789112424.56 total 120.10 last 71.53)
total: 10000 open/close in 123.30 seconds: 81.10 ops/second
 - unlinked 0 (time 1789112433 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 780 (time 1789112528.88 total 10.00 last 77.97)
 - open/close 1508 (time 1789112538.89 total 20.01 last 72.75)
 - open/close 2126 (time 1789112548.89 total 30.01 last 61.79)
 - open/close 2860 (time 1789112558.90 total 40.02 last 73.34)
 - open/close 3615 (time 1789112568.91 total 50.04 last 75.38)
 - open/close 4478 (time 1789112578.92 total 60.04 last 86.28)
 - open/close 5236 (time 1789112588.92 total 70.05 last 75.74)
 - open/close 5886 (time 1789112598.94 total 80.07 last 64.88)
 - open/close 6608 (time 1789112608.96 total 90.08 last 72.09)
 - open/close 7347 (time 1789112618.97 total 100.09 last 73.83)
 - open/close 8013 (time 1789112628.98 total 110.10 last 66.54)
 - open/close 8839 (time 1789112638.98 total 120.10 last 82.60)
 - open/close 9749 (time 1789112648.98 total 130.10 last 90.99)
total: 10000 open/close in 132.65 seconds: 75.39 ops/second
 - unlinked 0 (time 1789112656 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 705 (time 1789112752.03 total 10.02 last 70.34)
 - open/close 1549 (time 1789112762.03 total 20.03 last 84.38)
 - open/close 2381 (time 1789112772.04 total 30.03 last 83.15)
 - open/close 3252 (time 1789112782.04 total 40.04 last 87.06)
 - open/close 4021 (time 1789112792.05 total 50.05 last 76.83)
 - open/close 4760 (time 1789112802.05 total 60.05 last 73.89)
 - open/close 5515 (time 1789112812.06 total 70.05 last 75.48)
 - open/close 6207 (time 1789112822.06 total 80.06 last 69.15)
 - open/close 7082 (time 1789112832.07 total 90.06 last 87.44)
 - open/close 7892 (time 1789112842.08 total 100.07 last 80.93)
 - open/close 8694 (time 1789112852.09 total 110.08 last 80.13)
 - open/close 9430 (time 1789112862.09 total 120.09 last 73.57)
total: 10000 open/close in 126.93 seconds: 78.78 ops/second
 - unlinked 0 (time 1789112873 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 825 (time 1789112970.36 total 10.01 last 82.45)
 - open/close 1742 (time 1789112980.37 total 20.02 last 91.59)
 - open/close 2656 (time 1789112990.37 total 30.02 last 91.39)
 - open/close 3548 (time 1789113000.37 total 40.02 last 89.19)
 - open/close 4406 (time 1789113010.39 total 50.03 last 85.68)
 - open/close 5156 (time 1789113020.39 total 60.04 last 74.97)
 - open/close 5915 (time 1789113030.39 total 70.04 last 75.89)
 - open/close 6743 (time 1789113040.40 total 80.05 last 82.75)
 - open/close 7513 (time 1789113050.41 total 90.06 last 76.91)
 - open/close 8335 (time 1789113060.41 total 100.06 last 82.19)
 - open/close 9143 (time 1789113070.42 total 110.06 last 80.76)
total: 10000 open/close in 120.02 seconds: 83.32 ops/second
 - unlinked 0 (time 1789113083 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
On OST0, 12230 used inodes rc=0
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2