== conf-sanity test 69: replace an OST with the same index ========================================================== 09:19:54 (1788441594)
start mds service on oleg247-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg247-server'
oleg247-server: oleg247-server.virtnet: executing load_modules_local
oleg247-server: Loading modules from /home/green/git/lustre-release/lustre
oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg247-server: detected 4 online CPUs by sysfs
oleg247-server: MODOPTS_LIBCFS=
oleg247-server: Force libcfs to create 2 CPU partitions
oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg247-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg247-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1
Started lustre-OST0000
oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg247-client.virtnet:  -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 594 (time 1788441686.96 total 10.02 last 59.29)
 - open/close 1201 (time 1788441696.97 total 20.03 last 60.64)
 - open/close 1908 (time 1788441706.98 total 30.04 last 70.64)
 - open/close 2522 (time 1788441716.98 total 40.04 last 61.40)
 - open/close 3168 (time 1788441726.99 total 50.04 last 64.59)
 - open/close 3816 (time 1788441736.99 total 60.04 last 64.76)
 - open/close 4464 (time 1788441747.01 total 70.07 last 64.66)
 - open/close 5056 (time 1788441757.02 total 80.08 last 59.14)
 - open/close 5795 (time 1788441767.03 total 90.08 last 73.84)
 - open/close 6448 (time 1788441777.03 total 100.09 last 65.29)
 - open/close 7165 (time 1788441787.04 total 110.09 last 71.65)
 - open/close 7741 (time 1788441797.07 total 120.12 last 57.42)
 - open/close 8379 (time 1788441807.08 total 130.14 last 63.73)
 - open/close 8968 (time 1788441817.08 total 140.14 last 58.90)
 - open/close 9586 (time 1788441827.09 total 150.15 last 61.74)
total: 10000 open/close in 156.93 seconds: 63.72 ops/second
 - unlinked 0 (time 1788441838 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 593 (time 1788441956.88 total 10.00 last 59.28)
 - open/close 1237 (time 1788441966.89 total 20.01 last 64.38)
 - open/close 1866 (time 1788441976.89 total 30.01 last 62.90)
 - open/close 2430 (time 1788441986.89 total 40.01 last 56.36)
 - open/close 3051 (time 1788441996.90 total 50.02 last 62.07)
 - open/close 3713 (time 1788442006.93 total 60.05 last 66.00)
 - open/close 4422 (time 1788442016.94 total 70.06 last 70.85)
 - open/close 5061 (time 1788442026.95 total 80.07 last 63.81)
 - open/close 5760 (time 1788442036.95 total 90.07 last 69.88)
 - open/close 6482 (time 1788442046.96 total 100.08 last 72.15)
 - open/close 7152 (time 1788442056.98 total 110.10 last 66.88)
 - open/close 7882 (time 1788442066.98 total 120.10 last 72.96)
 - open/close 8575 (time 1788442076.99 total 130.11 last 69.27)
 - open/close 9210 (time 1788442087.00 total 140.12 last 63.44)
 - open/close 9868 (time 1788442097.01 total 150.13 last 65.72)
total: 10000 open/close in 152.23 seconds: 65.69 ops/second
 - unlinked 0 (time 1788442104 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 605 (time 1788442215.42 total 10.01 last 60.41)
 - open/close 1142 (time 1788442225.43 total 20.02 last 53.68)
 - open/close 1778 (time 1788442235.44 total 30.03 last 63.51)
 - open/close 2482 (time 1788442245.44 total 40.03 last 70.39)
 - open/close 3153 (time 1788442255.45 total 50.04 last 67.05)
 - open/close 3779 (time 1788442265.47 total 60.06 last 62.49)
 - open/close 4524 (time 1788442275.48 total 70.07 last 74.42)
 - open/close 5219 (time 1788442285.49 total 80.08 last 69.43)
 - open/close 5883 (time 1788442295.50 total 90.09 last 66.33)
 - open/close 6540 (time 1788442305.51 total 100.10 last 65.66)
 - open/close 7256 (time 1788442315.52 total 110.11 last 71.53)
 - open/close 7961 (time 1788442325.52 total 120.11 last 70.50)
 - open/close 8644 (time 1788442335.52 total 130.11 last 68.30)
 - open/close 9283 (time 1788442345.53 total 140.12 last 63.81)
 - open/close 9961 (time 1788442355.54 total 150.13 last 67.75)
total: 10000 open/close in 150.99 seconds: 66.23 ops/second
 - unlinked 0 (time 1788442361 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 639 (time 1788442476.39 total 10.01 last 63.86)
 - open/close 1346 (time 1788442486.40 total 20.02 last 70.62)
 - open/close 2065 (time 1788442496.40 total 30.02 last 71.89)
 - open/close 2871 (time 1788442506.42 total 40.03 last 80.49)
 - open/close 3583 (time 1788442516.44 total 50.05 last 71.07)
 - open/close 4285 (time 1788442526.44 total 60.05 last 70.19)
 - open/close 5031 (time 1788442536.45 total 70.06 last 74.53)
 - open/close 5744 (time 1788442546.45 total 80.07 last 71.26)
 - open/close 6419 (time 1788442556.46 total 90.07 last 67.47)
 - open/close 7123 (time 1788442566.47 total 100.08 last 70.35)
 - open/close 7792 (time 1788442576.48 total 110.09 last 66.83)
 - open/close 8418 (time 1788442586.48 total 120.10 last 62.55)
 - open/close 9175 (time 1788442596.49 total 130.10 last 75.66)
 - open/close 9915 (time 1788442606.51 total 140.12 last 73.84)
total: 10000 open/close in 141.24 seconds: 70.80 ops/second
 - unlinked 0 (time 1788442611 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 713 (time 1788442721.17 total 10.03 last 71.11)
 - open/close 1349 (time 1788442731.18 total 20.04 last 63.54)
 - open/close 1967 (time 1788442741.18 total 30.04 last 61.78)
 - open/close 2612 (time 1788442751.19 total 40.05 last 64.42)
 - open/close 3286 (time 1788442761.20 total 50.05 last 67.38)
 - open/close 3956 (time 1788442771.20 total 60.06 last 66.96)
 - open/close 4648 (time 1788442781.21 total 70.07 last 69.14)
 - open/close 5360 (time 1788442791.22 total 80.08 last 71.12)
 - open/close 6055 (time 1788442801.23 total 90.09 last 69.46)
 - open/close 6795 (time 1788442811.24 total 100.10 last 73.91)
 - open/close 7562 (time 1788442821.25 total 110.11 last 76.65)
 - open/close 8294 (time 1788442831.25 total 120.11 last 73.18)
 - open/close 8967 (time 1788442841.26 total 130.12 last 67.22)
 - open/close 9673 (time 1788442851.27 total 140.12 last 70.58)
total: 10000 open/close in 145.55 seconds: 68.70 ops/second
 - unlinked 0 (time 1788442861 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 718 (time 1788442968.23 total 10.00 last 71.79)
 - open/close 1417 (time 1788442978.23 total 20.00 last 69.89)
 - open/close 2090 (time 1788442988.23 total 30.01 last 67.26)
 - open/close 2779 (time 1788442998.24 total 40.01 last 68.88)
 - open/close 3420 (time 1788443008.25 total 50.02 last 64.03)
 - open/close 4074 (time 1788443018.26 total 60.03 last 65.35)
 - open/close 4660 (time 1788443028.26 total 70.03 last 58.58)
 - open/close 5295 (time 1788443038.26 total 80.04 last 63.46)
 - open/close 5827 (time 1788443048.28 total 90.05 last 53.14)
 - open/close 6441 (time 1788443058.28 total 100.05 last 61.38)
 - open/close 7143 (time 1788443068.29 total 110.06 last 70.15)
 - open/close 7768 (time 1788443078.30 total 120.07 last 62.41)
 - open/close 8412 (time 1788443088.31 total 130.08 last 64.37)
 - open/close 9077 (time 1788443098.31 total 140.08 last 66.50)
 - open/close 9747 (time 1788443108.31 total 150.09 last 66.96)
total: 10000 open/close in 153.22 seconds: 65.27 ops/second
 - unlinked 0 (time 1788443116 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 572 (time 1788443231.62 total 10.00 last 57.18)
 - open/close 1187 (time 1788443241.64 total 20.02 last 61.39)
 - open/close 1863 (time 1788443251.65 total 30.03 last 67.53)
 - open/close 2379 (time 1788443261.66 total 40.04 last 51.57)
 - open/close 3001 (time 1788443271.66 total 50.04 last 62.18)
 - open/close 3729 (time 1788443281.67 total 60.05 last 72.75)
 - open/close 4418 (time 1788443291.69 total 70.07 last 68.75)
 - open/close 5001 (time 1788443301.69 total 80.07 last 58.29)
 - open/close 5558 (time 1788443311.69 total 90.07 last 55.70)
 - open/close 6266 (time 1788443321.70 total 100.08 last 70.71)
 - open/close 6896 (time 1788443331.71 total 110.09 last 62.95)
 - open/close 7555 (time 1788443341.72 total 120.11 last 65.81)
 - open/close 8354 (time 1788443351.73 total 130.11 last 79.84)
 - open/close 9046 (time 1788443361.74 total 140.12 last 69.17)
 - open/close 9680 (time 1788443371.74 total 150.12 last 63.39)
total: 10000 open/close in 154.64 seconds: 64.66 ops/second
 - unlinked 0 (time 1788443380 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 543 (time 1788443497.50 total 10.01 last 54.25)
 - open/close 1128 (time 1788443507.51 total 20.02 last 58.41)
 - open/close 1604 (time 1788443517.53 total 30.04 last 47.53)
 - open/close 2122 (time 1788443527.55 total 40.06 last 51.68)
 - open/close 2863 (time 1788443537.55 total 50.06 last 74.10)
 - open/close 3596 (time 1788443547.55 total 60.07 last 73.27)
 - open/close 4331 (time 1788443557.56 total 70.07 last 73.44)
 - open/close 4947 (time 1788443567.57 total 80.08 last 61.54)
 - open/close 5629 (time 1788443577.58 total 90.10 last 68.11)
 - open/close 6258 (time 1788443587.61 total 100.12 last 62.77)
 - open/close 6981 (time 1788443597.61 total 110.12 last 72.24)
 - open/close 7626 (time 1788443607.61 total 120.13 last 64.49)
 - open/close 8303 (time 1788443617.62 total 130.13 last 67.65)
 - open/close 9024 (time 1788443627.65 total 140.16 last 71.91)
 - open/close 9749 (time 1788443637.65 total 150.16 last 72.50)
total: 10000 open/close in 152.95 seconds: 65.38 ops/second
 - unlinked 0 (time 1788443645 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 430 (time 1788443761.19 total 10.02 last 42.91)
 - open/close 1130 (time 1788443771.20 total 20.03 last 69.94)
 - open/close 1827 (time 1788443781.22 total 30.05 last 69.58)
 - open/close 2521 (time 1788443791.24 total 40.06 last 69.28)
 - open/close 3123 (time 1788443801.25 total 50.08 last 60.08)
 - open/close 3656 (time 1788443811.26 total 60.09 last 53.29)
 - open/close 4238 (time 1788443821.26 total 70.09 last 58.19)
 - open/close 4721 (time 1788443831.26 total 80.09 last 48.29)
 - open/close 5250 (time 1788443841.27 total 90.10 last 52.85)
 - open/close 5759 (time 1788443851.28 total 100.11 last 50.84)
 - open/close 6298 (time 1788443861.30 total 110.13 last 53.81)
 - open/close 7021 (time 1788443871.31 total 120.13 last 72.25)
 - open/close 7857 (time 1788443881.31 total 130.14 last 83.53)
 - open/close 8460 (time 1788443891.32 total 140.15 last 60.27)
 - open/close 9251 (time 1788443901.33 total 150.16 last 78.98)
 - open/close 9888 (time 1788443911.34 total 160.17 last 63.67)
total: 10000 open/close in 161.87 seconds: 61.78 ops/second
 - unlinked 0 (time 1788443918 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 601 (time 1788444039.51 total 10.00 last 60.09)
 - open/close 1237 (time 1788444049.52 total 20.01 last 63.56)
 - open/close 1801 (time 1788444059.52 total 30.02 last 56.35)
 - open/close 2571 (time 1788444069.55 total 40.04 last 76.82)
 - open/close 3198 (time 1788444079.57 total 50.06 last 62.56)
 - open/close 3817 (time 1788444089.58 total 60.07 last 61.87)
 - open/close 4603 (time 1788444099.58 total 70.07 last 78.56)
 - open/close 5386 (time 1788444109.59 total 80.08 last 78.24)
 - open/close 6029 (time 1788444119.59 total 90.09 last 64.27)
 - open/close 6639 (time 1788444129.59 total 100.09 last 61.00)
 - open/close 7381 (time 1788444139.60 total 110.09 last 74.17)
 - open/close 7900 (time 1788444149.64 total 120.13 last 51.67)
 - open/close 8601 (time 1788444159.67 total 130.16 last 69.89)
 - open/close 9285 (time 1788444169.69 total 140.18 last 68.30)
total: 10000 open/close in 149.51 seconds: 66.88 ops/second
 - unlinked 0 (time 1788444184 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg247-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg247-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.147@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.147@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg247-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1
Started lustre-OST0000
oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg247-client.virtnet:  -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg247-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg247-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.147@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.147@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.147@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.147@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.147@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg247-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1
Started lustre-OST0000
oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg247-client.virtnet:  -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg247-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg247-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server
stop mds service on oleg247-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server

unloading modules via unload_modules_local on: 'oleg247-server'
oleg247-server: oleg247-server.virtnet: executing unload_modules_local
oleg247-server: 
modules unloaded.
pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2
pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2
pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2