== conf-sanity test 69: replace an OST with the same index ========================================================== 19:44:20 (1786751060)
start mds service on oleg105-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg105-server'
oleg105-server: oleg105-server.virtnet: executing load_modules_local
oleg105-server: Loading modules from /home/green/git/lustre-release/lustre
oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg105-server: detected 4 online CPUs by sysfs
oleg105-server: MODOPTS_LIBCFS=
oleg105-server: Force libcfs to create 2 CPU partitions
oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg105-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-OST0000
oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg105-client.virtnet:  -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 845 (time 1786751156.25 total 10.00 last 84.47)
 - open/close 1644 (time 1786751166.26 total 20.00 last 79.89)
 - open/close 2460 (time 1786751176.27 total 30.02 last 81.51)
 - open/close 3198 (time 1786751186.27 total 40.02 last 73.74)
 - open/close 4015 (time 1786751196.29 total 50.03 last 81.60)
 - open/close 4825 (time 1786751206.29 total 60.04 last 80.97)
 - open/close 5701 (time 1786751216.29 total 70.04 last 87.55)
 - open/close 6523 (time 1786751226.30 total 80.04 last 82.20)
 - open/close 7330 (time 1786751236.30 total 90.05 last 80.66)
 - open/close 8223 (time 1786751246.31 total 100.06 last 89.23)
 - open/close 9053 (time 1786751256.31 total 110.06 last 82.96)
 - open/close 9935 (time 1786751266.32 total 120.06 last 88.17)
total: 10000 open/close in 120.88 seconds: 82.73 ops/second
 - unlinked 0 (time 1786751271 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 804 (time 1786751371.05 total 10.01 last 80.30)
 - open/close 1614 (time 1786751381.07 total 20.03 last 80.88)
 - open/close 2392 (time 1786751391.07 total 30.04 last 77.73)
 - open/close 3162 (time 1786751401.08 total 40.04 last 76.98)
 - open/close 3999 (time 1786751411.09 total 50.05 last 83.63)
 - open/close 4723 (time 1786751421.09 total 60.05 last 72.39)
 - open/close 5515 (time 1786751431.09 total 70.05 last 79.19)
 - open/close 6304 (time 1786751441.11 total 80.07 last 78.72)
 - open/close 7075 (time 1786751451.12 total 90.08 last 77.07)
 - open/close 7863 (time 1786751461.12 total 100.08 last 78.74)
 - open/close 8663 (time 1786751471.13 total 110.09 last 79.97)
 - open/close 9478 (time 1786751481.14 total 120.10 last 81.43)
total: 10000 open/close in 126.56 seconds: 79.01 ops/second
 - unlinked 0 (time 1786751492 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 606 (time 1786751598.91 total 10.01 last 60.55)
 - open/close 1211 (time 1786751608.92 total 20.01 last 60.46)
 - open/close 2039 (time 1786751618.92 total 30.02 last 82.79)
 - open/close 2727 (time 1786751628.93 total 40.02 last 68.75)
 - open/close 3503 (time 1786751638.93 total 50.02 last 77.59)
 - open/close 4296 (time 1786751648.93 total 60.03 last 79.28)
 - open/close 5056 (time 1786751658.94 total 70.03 last 75.95)
 - open/close 5788 (time 1786751668.95 total 80.04 last 73.14)
 - open/close 6459 (time 1786751678.95 total 90.04 last 67.09)
 - open/close 7224 (time 1786751688.95 total 100.04 last 76.48)
 - open/close 7834 (time 1786751698.95 total 110.05 last 60.98)
 - open/close 8525 (time 1786751708.96 total 120.05 last 69.06)
 - open/close 9461 (time 1786751718.97 total 130.06 last 93.51)
total: 10000 open/close in 138.17 seconds: 72.38 ops/second
 - unlinked 0 (time 1786751731 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 804 (time 1786751826.61 total 10.00 last 80.36)
 - open/close 1590 (time 1786751836.61 total 20.01 last 78.59)
 - open/close 2464 (time 1786751846.62 total 30.02 last 87.31)
 - open/close 3288 (time 1786751856.62 total 40.02 last 82.35)
 - open/close 4083 (time 1786751866.63 total 50.03 last 79.42)
 - open/close 4714 (time 1786751876.63 total 60.03 last 63.10)
 - open/close 5451 (time 1786751886.64 total 70.04 last 73.68)
 - open/close 6301 (time 1786751896.64 total 80.04 last 84.95)
 - open/close 7119 (time 1786751906.64 total 90.04 last 81.80)
 - open/close 7831 (time 1786751916.66 total 100.06 last 71.09)
 - open/close 8519 (time 1786751926.66 total 110.06 last 68.79)
 - open/close 9313 (time 1786751936.68 total 120.08 last 79.20)
total: 10000 open/close in 128.42 seconds: 77.87 ops/second
 - unlinked 0 (time 1786751950 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 804 (time 1786752045.23 total 10.00 last 80.40)
 - open/close 1580 (time 1786752055.23 total 20.00 last 77.57)
 - open/close 2323 (time 1786752065.25 total 30.02 last 74.16)
 - open/close 3013 (time 1786752075.26 total 40.03 last 68.96)
 - open/close 3993 (time 1786752085.26 total 50.03 last 97.97)
 - open/close 4890 (time 1786752095.27 total 60.04 last 89.64)
 - open/close 5622 (time 1786752105.27 total 70.04 last 73.15)
 - open/close 6407 (time 1786752115.28 total 80.05 last 78.47)
 - open/close 7197 (time 1786752125.28 total 90.05 last 78.97)
 - open/close 8090 (time 1786752135.31 total 100.08 last 89.05)
 - open/close 8909 (time 1786752145.32 total 110.09 last 81.86)
 - open/close 9883 (time 1786752155.33 total 120.10 last 97.30)
total: 10000 open/close in 121.71 seconds: 82.16 ops/second
 - unlinked 0 (time 1786752161 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 789 (time 1786752258.50 total 10.00 last 78.90)
 - open/close 1552 (time 1786752268.50 total 20.00 last 76.28)
 - open/close 2368 (time 1786752278.50 total 30.01 last 81.55)
 - open/close 3164 (time 1786752288.51 total 40.01 last 79.56)
 - open/close 3874 (time 1786752298.53 total 50.03 last 70.85)
 - open/close 4543 (time 1786752308.53 total 60.04 last 66.88)
 - open/close 5238 (time 1786752318.54 total 70.05 last 69.42)
 - open/close 5910 (time 1786752328.55 total 80.06 last 67.14)
 - open/close 6663 (time 1786752338.56 total 90.06 last 75.28)
 - open/close 7395 (time 1786752348.56 total 100.07 last 73.17)
 - open/close 8188 (time 1786752358.56 total 110.07 last 79.27)
 - open/close 9032 (time 1786752368.57 total 120.07 last 84.37)
 - open/close 9748 (time 1786752378.58 total 130.09 last 71.49)
total: 10000 open/close in 133.51 seconds: 74.90 ops/second
 - unlinked 0 (time 1786752387 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 553 (time 1786752493.49 total 10.01 last 55.25)
 - open/close 1275 (time 1786752503.50 total 20.02 last 72.11)
 - open/close 1910 (time 1786752513.52 total 30.04 last 63.36)
 - open/close 2495 (time 1786752523.53 total 40.05 last 58.45)
 - open/close 3122 (time 1786752533.54 total 50.06 last 62.64)
 - open/close 3793 (time 1786752543.59 total 60.11 last 66.78)
 - open/close 4442 (time 1786752553.60 total 70.12 last 64.83)
 - open/close 5155 (time 1786752563.61 total 80.13 last 71.23)
 - open/close 5910 (time 1786752573.61 total 90.13 last 75.47)
 - open/close 6558 (time 1786752583.62 total 100.14 last 64.73)
 - open/close 7206 (time 1786752593.63 total 110.15 last 64.73)
 - open/close 8046 (time 1786752603.63 total 120.16 last 83.98)
 - open/close 8837 (time 1786752613.64 total 130.16 last 79.06)
 - open/close 9633 (time 1786752623.68 total 140.20 last 79.31)
total: 10000 open/close in 146.82 seconds: 68.11 ops/second
 - unlinked 0 (time 1786752636 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 731 (time 1786752731.63 total 10.01 last 73.02)
 - open/close 1539 (time 1786752741.64 total 20.02 last 80.72)
 - open/close 2235 (time 1786752751.66 total 30.04 last 69.47)
 - open/close 3086 (time 1786752761.66 total 40.05 last 85.03)
 - open/close 4016 (time 1786752771.67 total 50.06 last 92.93)
 - open/close 4838 (time 1786752781.68 total 60.07 last 82.11)
 - open/close 5633 (time 1786752791.69 total 70.08 last 79.42)
 - open/close 6304 (time 1786752801.71 total 80.09 last 67.01)
 - open/close 6879 (time 1786752811.73 total 90.12 last 57.35)
 - open/close 7722 (time 1786752821.74 total 100.12 last 84.24)
 - open/close 8599 (time 1786752831.74 total 110.12 last 87.70)
 - open/close 9429 (time 1786752841.75 total 120.14 last 82.87)
total: 10000 open/close in 128.56 seconds: 77.78 ops/second
 - unlinked 0 (time 1786752854 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 739 (time 1786752955.78 total 10.02 last 73.77)
 - open/close 1529 (time 1786752965.79 total 20.04 last 78.85)
 - open/close 2233 (time 1786752975.82 total 30.06 last 70.22)
 - open/close 3012 (time 1786752985.83 total 40.07 last 77.85)
 - open/close 3836 (time 1786752995.83 total 50.07 last 82.38)
 - open/close 4697 (time 1786753005.84 total 60.08 last 86.03)
 - open/close 5428 (time 1786753015.84 total 70.08 last 73.06)
 - open/close 6261 (time 1786753025.85 total 80.09 last 83.21)
 - open/close 6976 (time 1786753035.86 total 90.10 last 71.46)
 - open/close 7684 (time 1786753045.86 total 100.10 last 70.78)
 - open/close 8412 (time 1786753055.87 total 110.11 last 72.75)
 - open/close 9195 (time 1786753065.88 total 120.13 last 78.16)
total: 10000 open/close in 129.76 seconds: 77.07 ops/second
 - unlinked 0 (time 1786753078 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 896 (time 1786753172.80 total 10.00 last 89.59)
 - open/close 1803 (time 1786753182.81 total 20.02 last 90.57)
 - open/close 2693 (time 1786753192.82 total 30.02 last 88.93)
 - open/close 3551 (time 1786753202.82 total 40.03 last 85.76)
 - open/close 4391 (time 1786753212.83 total 50.03 last 83.98)
 - open/close 5241 (time 1786753222.83 total 60.04 last 84.95)
 - open/close 6066 (time 1786753232.84 total 70.04 last 82.46)
 - open/close 6840 (time 1786753242.84 total 80.04 last 77.38)
 - open/close 7656 (time 1786753252.85 total 90.05 last 81.55)
 - open/close 8543 (time 1786753262.85 total 100.06 last 88.65)
 - open/close 9356 (time 1786753272.85 total 110.06 last 81.28)
total: 10000 open/close in 117.67 seconds: 84.98 ops/second
 - unlinked 0 (time 1786753284 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg105-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg105-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.105@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.105@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg105-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-OST0000
oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg105-server: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg105-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg105-client.virtnet:  -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg105-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg105-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.105@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.105@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.105@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.105@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.105@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg105-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-OST0000
oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg105-server: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg105-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg105-client.virtnet:  -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg105-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg105-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server
stop mds service on oleg105-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server

unloading modules via unload_modules_local on: 'oleg105-server'
oleg105-server: oleg105-server.virtnet: executing unload_modules_local
oleg105-server: 
modules unloaded.
pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2
pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2