== conf-sanity test 69: replace an OST with the same index ========================================================== 05:14:01 (1778922841)
start mds service on oleg203-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg203-server'
oleg203-server: oleg203-server.virtnet: executing load_modules_local
oleg203-server: Loading modules from /home/green/git/lustre-release/lustre
oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg203-server: detected 4 online CPUs by sysfs
oleg203-server: MODOPTS_LIBCFS=
oleg203-server: Force libcfs to create 2 CPU partitions
oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 812 (time 1778922919.57 total 10.00 last 81.19)
 - open/close 1463 (time 1778922929.57 total 20.00 last 65.09)
 - open/close 2348 (time 1778922939.58 total 30.01 last 88.43)
 - open/close 3210 (time 1778922949.60 total 40.03 last 86.03)
 - open/close 3941 (time 1778922959.61 total 50.05 last 72.99)
 - open/close 4632 (time 1778922969.62 total 60.05 last 69.09)
 - open/close 5468 (time 1778922979.62 total 70.05 last 83.55)
 - open/close 6347 (time 1778922989.63 total 80.06 last 87.87)
 - open/close 7191 (time 1778922999.64 total 90.07 last 84.32)
 - open/close 8030 (time 1778923009.65 total 100.08 last 83.77)
 - open/close 8717 (time 1778923019.66 total 110.09 last 68.64)
 - open/close 9599 (time 1778923029.67 total 120.10 last 88.11)
total: 10000 open/close in 124.54 seconds: 80.29 ops/second
 - unlinked 0 (time 1778923038 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 507 (time 1778923128.84 total 10.00 last 50.68)
 - open/close 1122 (time 1778923138.85 total 20.01 last 61.46)
 - open/close 1785 (time 1778923148.85 total 30.01 last 66.29)
 - open/close 2409 (time 1778923158.86 total 40.02 last 62.34)
 - open/close 3186 (time 1778923168.86 total 50.02 last 77.69)
 - open/close 4078 (time 1778923178.87 total 60.03 last 89.16)
 - open/close 4942 (time 1778923188.87 total 70.03 last 86.37)
 - open/close 5831 (time 1778923198.88 total 80.04 last 88.84)
 - open/close 6750 (time 1778923208.89 total 90.05 last 91.77)
 - open/close 7631 (time 1778923218.90 total 100.06 last 88.05)
 - open/close 8452 (time 1778923228.90 total 110.06 last 82.10)
 - open/close 9315 (time 1778923238.90 total 120.06 last 86.27)
total: 10000 open/close in 127.44 seconds: 78.47 ops/second
 - unlinked 0 (time 1778923249 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 859 (time 1778923342.59 total 10.01 last 85.85)
 - open/close 1714 (time 1778923352.60 total 20.02 last 85.42)
 - open/close 2550 (time 1778923362.60 total 30.02 last 83.53)
 - open/close 3165 (time 1778923372.61 total 40.03 last 61.45)
 - open/close 3979 (time 1778923382.62 total 50.04 last 81.35)
 - open/close 4975 (time 1778923392.62 total 60.04 last 99.58)
 - open/close 5836 (time 1778923402.63 total 70.05 last 86.05)
 - open/close 6659 (time 1778923412.63 total 80.05 last 82.28)
 - open/close 7493 (time 1778923422.63 total 90.05 last 83.39)
 - open/close 8373 (time 1778923432.64 total 100.06 last 87.96)
 - open/close 9150 (time 1778923442.64 total 110.06 last 77.66)
total: 10000 open/close in 118.66 seconds: 84.28 ops/second
 - unlinked 0 (time 1778923456 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 777 (time 1778923551.64 total 10.01 last 77.63)
 - open/close 1759 (time 1778923561.65 total 20.02 last 98.09)
 - open/close 2628 (time 1778923571.65 total 30.02 last 86.90)
 - open/close 3367 (time 1778923581.66 total 40.03 last 73.82)
 - open/close 4128 (time 1778923591.67 total 50.05 last 75.99)
 - open/close 4934 (time 1778923601.68 total 60.05 last 80.59)
 - open/close 5695 (time 1778923611.68 total 70.05 last 76.08)
 - open/close 6410 (time 1778923621.68 total 80.05 last 71.49)
 - open/close 7029 (time 1778923631.68 total 90.05 last 61.89)
 - open/close 7863 (time 1778923641.69 total 100.06 last 83.36)
 - open/close 8683 (time 1778923651.69 total 110.06 last 82.00)
 - open/close 9454 (time 1778923661.69 total 120.06 last 77.08)
total: 10000 open/close in 125.87 seconds: 79.45 ops/second
 - unlinked 0 (time 1778923672 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 933 (time 1778923766.48 total 10.00 last 93.27)
 - open/close 1682 (time 1778923776.49 total 20.01 last 74.88)
 - open/close 2550 (time 1778923786.49 total 30.01 last 86.78)
 - open/close 3280 (time 1778923796.50 total 40.02 last 72.95)
 - open/close 3983 (time 1778923806.51 total 50.03 last 70.22)
 - open/close 4661 (time 1778923816.51 total 60.03 last 67.79)
 - open/close 5552 (time 1778923826.51 total 70.03 last 89.07)
 - open/close 6303 (time 1778923836.53 total 80.05 last 75.00)
 - open/close 7088 (time 1778923846.53 total 90.05 last 78.49)
 - open/close 7798 (time 1778923856.53 total 100.05 last 70.99)
 - open/close 8614 (time 1778923866.54 total 110.06 last 81.51)
 - open/close 9382 (time 1778923876.55 total 120.06 last 76.77)
total: 10000 open/close in 129.54 seconds: 77.19 ops/second
 - unlinked 0 (time 1778923891 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 643 (time 1778923997.76 total 10.01 last 64.24)
 - open/close 1372 (time 1778924007.76 total 20.02 last 72.84)
 - open/close 2220 (time 1778924017.77 total 30.03 last 84.73)
 - open/close 2949 (time 1778924027.78 total 40.03 last 72.84)
 - open/close 3683 (time 1778924037.79 total 50.04 last 73.36)
 - open/close 4544 (time 1778924047.80 total 60.06 last 85.96)
 - open/close 5156 (time 1778924057.81 total 70.07 last 61.13)
 - open/close 5869 (time 1778924067.82 total 80.07 last 71.25)
 - open/close 6675 (time 1778924077.82 total 90.08 last 80.57)
 - open/close 7358 (time 1778924087.83 total 100.08 last 68.27)
 - open/close 8024 (time 1778924097.85 total 110.11 last 66.45)
 - open/close 8807 (time 1778924107.86 total 120.11 last 78.23)
 - open/close 9561 (time 1778924117.87 total 130.12 last 75.35)
total: 10000 open/close in 134.93 seconds: 74.11 ops/second
 - unlinked 0 (time 1778924127 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 637 (time 1778924233.97 total 10.00 last 63.70)
 - open/close 1385 (time 1778924243.98 total 20.01 last 74.74)
 - open/close 2069 (time 1778924253.98 total 30.01 last 68.40)
 - open/close 2770 (time 1778924263.98 total 40.01 last 70.07)
 - open/close 3512 (time 1778924273.98 total 50.01 last 74.20)
 - open/close 4127 (time 1778924283.98 total 60.01 last 61.50)
 - open/close 4760 (time 1778924293.98 total 70.02 last 63.29)
 - open/close 5240 (time 1778924304.00 total 80.03 last 47.94)
 - open/close 5904 (time 1778924314.00 total 90.03 last 66.38)
 - open/close 6703 (time 1778924324.01 total 100.04 last 79.83)
 - open/close 7442 (time 1778924334.02 total 110.05 last 73.83)
 - open/close 8289 (time 1778924344.02 total 120.05 last 84.69)
 - open/close 9006 (time 1778924354.03 total 130.06 last 71.63)
 - open/close 9552 (time 1778924364.03 total 140.07 last 54.56)
total: 10000 open/close in 149.47 seconds: 66.90 ops/second
 - unlinked 0 (time 1778924378 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 766 (time 1778924470.86 total 10.01 last 76.53)
 - open/close 1570 (time 1778924480.86 total 20.01 last 80.39)
 - open/close 2390 (time 1778924490.87 total 30.02 last 81.95)
 - open/close 3220 (time 1778924500.87 total 40.02 last 82.99)
 - open/close 3981 (time 1778924510.87 total 50.02 last 76.08)
 - open/close 4776 (time 1778924520.88 total 60.02 last 79.49)
 - open/close 5585 (time 1778924530.88 total 70.03 last 80.87)
 - open/close 6287 (time 1778924540.89 total 80.04 last 70.13)
 - open/close 7027 (time 1778924550.90 total 90.04 last 73.94)
 - open/close 7941 (time 1778924560.90 total 100.04 last 91.40)
 - open/close 8731 (time 1778924570.90 total 110.04 last 79.00)
 - open/close 9467 (time 1778924580.91 total 120.05 last 73.53)
total: 10000 open/close in 127.21 seconds: 78.61 ops/second
 - unlinked 0 (time 1778924592 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 739 (time 1778924689.12 total 10.03 last 73.68)
 - open/close 1422 (time 1778924699.13 total 20.05 last 68.20)
 - open/close 2424 (time 1778924709.13 total 30.05 last 100.18)
 - open/close 3305 (time 1778924719.14 total 40.05 last 88.08)
 - open/close 4101 (time 1778924729.14 total 50.05 last 79.58)
 - open/close 4893 (time 1778924739.16 total 60.08 last 79.02)
 - open/close 5659 (time 1778924749.17 total 70.08 last 76.57)
 - open/close 6382 (time 1778924759.17 total 80.08 last 72.29)
 - open/close 7154 (time 1778924769.17 total 90.08 last 77.19)
 - open/close 7940 (time 1778924779.18 total 100.09 last 78.54)
 - open/close 8653 (time 1778924789.18 total 110.09 last 71.29)
 - open/close 9488 (time 1778924799.19 total 120.10 last 83.41)
total: 10000 open/close in 126.43 seconds: 79.09 ops/second
 - unlinked 0 (time 1778924809 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 891 (time 1778924899.59 total 10.01 last 89.04)
 - open/close 1634 (time 1778924909.59 total 20.01 last 74.30)
 - open/close 2544 (time 1778924919.59 total 30.01 last 90.99)
 - open/close 3412 (time 1778924929.59 total 40.01 last 86.76)
 - open/close 4349 (time 1778924939.61 total 50.03 last 93.56)
 - open/close 5202 (time 1778924949.61 total 60.03 last 85.27)
 - open/close 6027 (time 1778924959.61 total 70.03 last 82.47)
 - open/close 6930 (time 1778924969.62 total 80.04 last 90.22)
 - open/close 7734 (time 1778924979.63 total 90.05 last 80.36)
 - open/close 8644 (time 1778924989.63 total 100.05 last 90.96)
 - open/close 9613 (time 1778924999.65 total 110.07 last 96.76)
total: 10000 open/close in 113.95 seconds: 87.76 ops/second
 - unlinked 0 (time 1778925006 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.103@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.103@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.103@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.103@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.103@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server
stop mds service on oleg203-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server

unloading modules via unload_modules_local on: 'oleg203-server'
oleg203-server: oleg203-server.virtnet: executing unload_modules_local
oleg203-server: 
modules unloaded.
pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2