== conf-sanity test 69: replace an OST with the same index ========================================================== 03:11:40 (1780384300)
start mds service on oleg429-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg429-server'
oleg429-server: oleg429-server.virtnet: executing load_modules_local
oleg429-server: Loading modules from /home/green/git/lustre-release/lustre
oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg429-server: detected 4 online CPUs by sysfs
oleg429-server: MODOPTS_LIBCFS=
oleg429-server: Force libcfs to create 2 CPU partitions
oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg429-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1
Started lustre-OST0000
oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg429-client.virtnet:  -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 791 (time 1780384384.64 total 10.01 last 78.99)
 - open/close 1495 (time 1780384394.65 total 20.03 last 70.31)
 - open/close 2220 (time 1780384404.65 total 30.03 last 72.47)
 - open/close 3001 (time 1780384414.66 total 40.04 last 78.03)
 - open/close 3746 (time 1780384424.67 total 50.05 last 74.44)
 - open/close 4446 (time 1780384434.67 total 60.05 last 69.97)
 - open/close 5152 (time 1780384444.68 total 70.06 last 70.54)
 - open/close 5879 (time 1780384454.69 total 80.06 last 72.67)
 - open/close 6607 (time 1780384464.69 total 90.07 last 72.79)
 - open/close 7416 (time 1780384474.69 total 100.07 last 80.88)
 - open/close 8205 (time 1780384484.71 total 110.08 last 78.78)
 - open/close 9027 (time 1780384494.72 total 120.09 last 82.13)
 - open/close 9789 (time 1780384504.72 total 130.09 last 76.19)
total: 10000 open/close in 132.76 seconds: 75.32 ops/second
 - unlinked 0 (time 1780384510 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 733 (time 1780384607.16 total 10.01 last 73.24)
 - open/close 1518 (time 1780384617.16 total 20.01 last 78.47)
 - open/close 2261 (time 1780384627.17 total 30.01 last 74.28)
 - open/close 3052 (time 1780384637.18 total 40.03 last 79.00)
 - open/close 3822 (time 1780384647.19 total 50.04 last 76.94)
 - open/close 4654 (time 1780384657.19 total 60.04 last 83.17)
 - open/close 5446 (time 1780384667.19 total 70.04 last 79.19)
 - open/close 6352 (time 1780384677.19 total 80.04 last 90.59)
 - open/close 7232 (time 1780384687.20 total 90.04 last 87.98)
 - open/close 8020 (time 1780384697.21 total 100.06 last 78.69)
 - open/close 8886 (time 1780384707.21 total 110.06 last 86.56)
 - open/close 9711 (time 1780384717.22 total 120.06 last 82.48)
total: 10000 open/close in 123.30 seconds: 81.10 ops/second
 - unlinked 0 (time 1780384723 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 823 (time 1780384810.40 total 10.01 last 82.23)
 - open/close 1666 (time 1780384820.41 total 20.02 last 84.22)
 - open/close 2429 (time 1780384830.41 total 30.02 last 76.27)
 - open/close 3346 (time 1780384840.42 total 40.03 last 91.63)
 - open/close 4100 (time 1780384850.42 total 50.03 last 75.37)
 - open/close 4927 (time 1780384860.43 total 60.04 last 82.64)
 - open/close 5808 (time 1780384870.44 total 70.05 last 88.04)
 - open/close 6619 (time 1780384880.44 total 80.05 last 81.05)
 - open/close 7569 (time 1780384890.44 total 90.06 last 94.98)
 - open/close 8612 (time 1780384900.45 total 100.06 last 104.25)
 - open/close 9693 (time 1780384910.46 total 110.07 last 107.96)
total: 10000 open/close in 113.50 seconds: 88.10 ops/second
 - unlinked 0 (time 1780384916 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 791 (time 1780385006.68 total 10.01 last 79.04)
 - open/close 1581 (time 1780385016.68 total 20.01 last 78.97)
 - open/close 2418 (time 1780385026.69 total 30.02 last 83.63)
 - open/close 3279 (time 1780385036.70 total 40.03 last 86.05)
 - open/close 4132 (time 1780385046.70 total 50.03 last 85.30)
 - open/close 5007 (time 1780385056.70 total 60.03 last 87.45)
 - open/close 5826 (time 1780385066.71 total 70.04 last 81.84)
 - open/close 6752 (time 1780385076.72 total 80.05 last 92.54)
 - open/close 7690 (time 1780385086.73 total 90.06 last 93.70)
 - open/close 8525 (time 1780385096.73 total 100.06 last 83.48)
 - open/close 9287 (time 1780385106.73 total 110.06 last 76.18)
total: 10000 open/close in 117.95 seconds: 84.78 ops/second
 - unlinked 0 (time 1780385117 ; total 0 ; last 0)
total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second
 - open/close 923 (time 1780385195.05 total 10.00 last 92.27)
 - open/close 1769 (time 1780385205.07 total 20.01 last 84.50)
 - open/close 2654 (time 1780385215.07 total 30.02 last 88.46)
 - open/close 3632 (time 1780385225.08 total 40.03 last 97.72)
 - open/close 4531 (time 1780385235.08 total 50.03 last 89.88)
 - open/close 5633 (time 1780385245.09 total 60.04 last 110.12)
 - open/close 6562 (time 1780385255.09 total 70.04 last 92.88)
 - open/close 7487 (time 1780385265.10 total 80.05 last 92.44)
 - open/close 8496 (time 1780385275.10 total 90.05 last 100.83)
 - open/close 9526 (time 1780385285.10 total 100.05 last 103.00)
total: 10000 open/close in 104.18 seconds: 95.99 ops/second
 - unlinked 0 (time 1780385291 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 474 (time 1780385367.13 total 10.03 last 47.27)
 - open/close 957 (time 1780385377.13 total 20.03 last 48.30)
 - open/close 1588 (time 1780385387.14 total 30.04 last 63.04)
 - open/close 2238 (time 1780385397.14 total 40.04 last 64.98)
 - open/close 3012 (time 1780385407.14 total 50.04 last 77.38)
 - open/close 3821 (time 1780385417.15 total 60.05 last 80.83)
 - open/close 4604 (time 1780385427.16 total 70.06 last 78.27)
 - open/close 5434 (time 1780385437.17 total 80.06 last 82.92)
 - open/close 6171 (time 1780385447.17 total 90.07 last 73.67)
 - open/close 6910 (time 1780385457.18 total 100.08 last 73.83)
 - open/close 7551 (time 1780385467.19 total 110.09 last 64.05)
 - open/close 8248 (time 1780385477.22 total 120.11 last 69.50)
 - open/close 9025 (time 1780385487.23 total 130.13 last 77.60)
 - open/close 9790 (time 1780385497.25 total 140.15 last 76.36)
total: 10000 open/close in 143.18 seconds: 69.84 ops/second
 - unlinked 0 (time 1780385504 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 796 (time 1780385607.09 total 10.01 last 79.50)
 - open/close 1468 (time 1780385617.10 total 20.02 last 67.16)
 - open/close 2220 (time 1780385627.10 total 30.02 last 75.19)
 - open/close 2956 (time 1780385637.10 total 40.02 last 73.59)
 - open/close 3586 (time 1780385647.11 total 50.03 last 62.96)
 - open/close 4156 (time 1780385657.11 total 60.03 last 56.98)
 - open/close 4843 (time 1780385667.12 total 70.04 last 68.64)
 - open/close 5486 (time 1780385677.13 total 80.05 last 64.22)
 - open/close 6162 (time 1780385687.13 total 90.05 last 67.59)
 - open/close 6871 (time 1780385697.14 total 100.07 last 70.82)
 - open/close 7655 (time 1780385707.15 total 110.07 last 78.38)
 - open/close 8408 (time 1780385717.16 total 120.08 last 75.23)
 - open/close 9180 (time 1780385727.17 total 130.09 last 77.10)
 - open/close 9943 (time 1780385737.18 total 140.10 last 76.21)
total: 10000 open/close in 140.78 seconds: 71.03 ops/second
 - unlinked 0 (time 1780385740 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 739 (time 1780385832.46 total 10.00 last 73.88)
 - open/close 1701 (time 1780385842.47 total 20.01 last 96.16)
 - open/close 2570 (time 1780385852.47 total 30.01 last 86.88)
 - open/close 3387 (time 1780385862.47 total 40.01 last 81.67)
 - open/close 4346 (time 1780385872.48 total 50.02 last 95.80)
 - open/close 5392 (time 1780385882.49 total 60.03 last 104.57)
 - open/close 6293 (time 1780385892.50 total 70.04 last 90.00)
 - open/close 7207 (time 1780385902.50 total 80.04 last 91.36)
 - open/close 8120 (time 1780385912.51 total 90.05 last 91.26)
 - open/close 9046 (time 1780385922.51 total 100.05 last 92.53)
total: 10000 open/close in 109.25 seconds: 91.54 ops/second
 - unlinked 0 (time 1780385934 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 1004 (time 1780386005.66 total 10.00 last 100.35)
 - open/close 1988 (time 1780386015.67 total 20.01 last 98.33)
 - open/close 3080 (time 1780386025.67 total 30.01 last 109.19)
 - open/close 4200 (time 1780386035.67 total 40.02 last 111.95)
 - open/close 5496 (time 1780386045.68 total 50.02 last 129.57)
 - open/close 6568 (time 1780386055.68 total 60.02 last 107.15)
 - open/close 7533 (time 1780386065.68 total 70.03 last 96.48)
 - open/close 8559 (time 1780386075.69 total 80.03 last 102.58)
 - open/close 9792 (time 1780386085.70 total 90.04 last 123.17)
total: 10000 open/close in 91.61 seconds: 109.16 ops/second
 - unlinked 0 (time 1780386089 ; total 0 ; last 0)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 1090 (time 1780386153.27 total 10.00 last 108.95)
 - open/close 2163 (time 1780386163.28 total 20.01 last 107.24)
 - open/close 3222 (time 1780386173.28 total 30.01 last 105.87)
 - open/close 4314 (time 1780386183.29 total 40.02 last 109.15)
 - open/close 5416 (time 1780386193.29 total 50.02 last 110.16)
 - open/close 6504 (time 1780386203.30 total 60.03 last 108.68)
 - open/close 7663 (time 1780386213.30 total 70.03 last 115.86)
 - open/close 8980 (time 1780386223.31 total 80.04 last 131.64)
total: 10000 open/close in 87.85 seconds: 113.83 ops/second
 - unlinked 0 (time 1780386233 ; total 0 ; last 0)
total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg429-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg429-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.129@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg429-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1
Started lustre-OST0000
oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg429-client.virtnet:  -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre
On OST0, 9454 used inodes rc=0
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg429-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg429-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.129@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.204.129@tcp sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.129@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg429-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1
Started lustre-OST0000
oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg429-client.virtnet:  -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg429-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg429-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server
stop mds service on oleg429-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server

unloading modules via unload_modules_local on: 'oleg429-server'
oleg429-server: oleg429-server.virtnet: executing unload_modules_local
oleg429-server: 
modules unloaded.
pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2
pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2
pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2