== conf-sanity test 69: replace an OST with the same index ========================================================== 16:54:47 (1786827287)
start mds service on oleg601-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg601-server'
oleg601-server: oleg601-server.virtnet: executing load_modules_local
oleg601-server: Loading modules from /home/green/git/lustre-release/lustre
oleg601-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg601-server: detected 4 online CPUs by sysfs
oleg601-server: MODOPTS_LIBCFS=
oleg601-server: Force libcfs to create 2 CPU partitions
oleg601-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg601-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg601-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg601-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1
Started lustre-OST0000
oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg601-client.virtnet:  -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 825 (time 1786827366.64 total 10.00 last 82.48)
 - open/close 1756 (time 1786827376.64 total 20.00 last 93.10)
 - open/close 2641 (time 1786827386.65 total 30.01 last 88.44)
 - open/close 3329 (time 1786827396.65 total 40.02 last 68.75)
 - open/close 4027 (time 1786827406.66 total 50.02 last 69.77)
 - open/close 4608 (time 1786827416.67 total 60.03 last 58.05)
 - open/close 5249 (time 1786827426.67 total 70.03 last 64.08)
 - open/close 5991 (time 1786827436.67 total 80.04 last 74.19)
 - open/close 6844 (time 1786827446.68 total 90.04 last 85.24)
 - open/close 7495 (time 1786827456.68 total 100.05 last 65.07)
 - open/close 8207 (time 1786827466.69 total 110.06 last 71.14)
 - open/close 8982 (time 1786827476.70 total 120.06 last 77.44)
 - open/close 9633 (time 1786827486.72 total 130.08 last 65.00)
total: 10000 open/close in 135.54 seconds: 73.78 ops/second
 - unlinked 0 (time 1786827495 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 867 (time 1786827602.06 total 10.01 last 86.60)
 - open/close 1733 (time 1786827612.06 total 20.01 last 86.57)
 - open/close 2576 (time 1786827622.07 total 30.02 last 84.25)
 - open/close 3402 (time 1786827632.07 total 40.02 last 82.59)
 - open/close 4198 (time 1786827642.08 total 50.03 last 79.56)
 - open/close 5056 (time 1786827652.08 total 60.03 last 85.74)
 - open/close 5917 (time 1786827662.09 total 70.04 last 86.05)
 - open/close 6839 (time 1786827672.09 total 80.04 last 92.20)
 - open/close 7734 (time 1786827682.09 total 90.04 last 89.48)
 - open/close 8642 (time 1786827692.09 total 100.04 last 90.79)
 - open/close 9547 (time 1786827702.09 total 110.04 last 90.50)
total: 10000 open/close in 115.30 seconds: 86.73 ops/second
 - unlinked 0 (time 1786827710 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 873 (time 1786827792.40 total 10.01 last 87.19)
 - open/close 1816 (time 1786827802.41 total 20.02 last 94.22)
 - open/close 2711 (time 1786827812.41 total 30.02 last 89.49)
 - open/close 3613 (time 1786827822.42 total 40.03 last 90.13)
 - open/close 4513 (time 1786827832.43 total 50.04 last 89.95)
 - open/close 5387 (time 1786827842.44 total 60.05 last 87.30)
 - open/close 6238 (time 1786827852.45 total 70.06 last 84.98)
 - open/close 7062 (time 1786827862.46 total 80.07 last 82.35)
 - open/close 7893 (time 1786827872.46 total 90.08 last 83.05)
 - open/close 8680 (time 1786827882.47 total 100.08 last 78.65)
 - open/close 9465 (time 1786827892.48 total 110.09 last 78.42)
total: 10000 open/close in 116.62 seconds: 85.75 ops/second
 - unlinked 0 (time 1786827902 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 709 (time 1786827990.52 total 10.01 last 70.84)
 - open/close 1365 (time 1786828000.53 total 20.02 last 65.49)
 - open/close 2040 (time 1786828010.54 total 30.03 last 67.46)
 - open/close 2880 (time 1786828020.55 total 40.04 last 83.94)
 - open/close 3631 (time 1786828030.57 total 50.06 last 74.96)
 - open/close 4301 (time 1786828040.57 total 60.06 last 66.98)
 - open/close 5160 (time 1786828050.57 total 70.06 last 85.89)
 - open/close 6013 (time 1786828060.57 total 80.06 last 85.30)
 - open/close 6898 (time 1786828070.57 total 90.07 last 88.44)
 - open/close 7809 (time 1786828080.58 total 100.07 last 91.06)
 - open/close 8717 (time 1786828090.59 total 110.08 last 90.72)
 - open/close 9603 (time 1786828100.59 total 120.08 last 88.56)
total: 10000 open/close in 124.41 seconds: 80.38 ops/second
 - unlinked 0 (time 1786828108 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 841 (time 1786828198.45 total 10.01 last 84.03)
 - open/close 1663 (time 1786828208.45 total 20.01 last 82.19)
 - open/close 2436 (time 1786828218.46 total 30.01 last 77.29)
 - open/close 3328 (time 1786828228.46 total 40.02 last 89.16)
 - open/close 4164 (time 1786828238.46 total 50.02 last 83.59)
 - open/close 4908 (time 1786828248.47 total 60.02 last 74.36)
 - open/close 5767 (time 1786828258.47 total 70.03 last 85.85)
 - open/close 6659 (time 1786828268.48 total 80.03 last 89.15)
 - open/close 7533 (time 1786828278.48 total 90.04 last 87.34)
 - open/close 8410 (time 1786828288.50 total 100.06 last 87.57)
 - open/close 9142 (time 1786828298.50 total 110.06 last 73.18)
 - open/close 9985 (time 1786828308.50 total 120.06 last 84.29)
total: 10000 open/close in 120.19 seconds: 83.20 ops/second
 - unlinked 0 (time 1786828313 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 877 (time 1786828399.10 total 10.01 last 87.62)
 - open/close 1773 (time 1786828409.11 total 20.02 last 89.53)
 - open/close 2684 (time 1786828419.11 total 30.02 last 91.10)
 - open/close 3631 (time 1786828429.11 total 40.02 last 94.66)
 - open/close 4521 (time 1786828439.12 total 50.03 last 88.95)
 - open/close 5436 (time 1786828449.13 total 60.04 last 91.38)
 - open/close 6330 (time 1786828459.13 total 70.04 last 89.38)
 - open/close 7282 (time 1786828469.14 total 80.05 last 95.12)
 - open/close 8186 (time 1786828479.15 total 90.06 last 90.37)
 - open/close 9110 (time 1786828489.16 total 100.07 last 92.30)
 - open/close 9997 (time 1786828499.16 total 110.07 last 88.66)
total: 10000 open/close in 110.09 seconds: 90.84 ops/second
 - unlinked 0 (time 1786828502 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 945 (time 1786828588.55 total 10.00 last 94.45)
 - open/close 1880 (time 1786828598.56 total 20.01 last 93.44)
 - open/close 2835 (time 1786828608.57 total 30.02 last 95.42)
 - open/close 3760 (time 1786828618.57 total 40.02 last 92.49)
 - open/close 4633 (time 1786828628.58 total 50.04 last 87.17)
 - open/close 5470 (time 1786828638.59 total 60.04 last 83.67)
 - open/close 6294 (time 1786828648.59 total 70.04 last 82.37)
 - open/close 7208 (time 1786828658.60 total 80.05 last 91.34)
 - open/close 8122 (time 1786828668.60 total 90.05 last 91.37)
 - open/close 8988 (time 1786828678.61 total 100.06 last 86.52)
 - open/close 9890 (time 1786828688.61 total 110.06 last 90.19)
total: 10000 open/close in 111.19 seconds: 89.93 ops/second
 - unlinked 0 (time 1786828694 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 846 (time 1786828779.19 total 10.00 last 84.58)
 - open/close 1701 (time 1786828789.19 total 20.01 last 85.46)
 - open/close 2617 (time 1786828799.20 total 30.02 last 91.48)
 - open/close 3557 (time 1786828809.21 total 40.02 last 93.97)
 - open/close 4525 (time 1786828819.21 total 50.02 last 96.79)
 - open/close 5447 (time 1786828829.21 total 60.03 last 92.15)
 - open/close 6360 (time 1786828839.22 total 70.03 last 91.26)
 - open/close 7292 (time 1786828849.22 total 80.04 last 93.14)
 - open/close 8164 (time 1786828859.23 total 90.05 last 87.11)
 - open/close 8876 (time 1786828869.25 total 100.07 last 71.07)
 - open/close 9635 (time 1786828879.25 total 110.07 last 75.90)
total: 10000 open/close in 114.90 seconds: 87.03 ops/second
 - unlinked 0 (time 1786828887 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 915 (time 1786828973.43 total 10.02 last 91.28)
 - open/close 1801 (time 1786828983.44 total 20.04 last 88.50)
 - open/close 2657 (time 1786828993.45 total 30.05 last 85.47)
 - open/close 3420 (time 1786829003.46 total 40.06 last 76.25)
 - open/close 4231 (time 1786829013.46 total 50.06 last 81.07)
 - open/close 5155 (time 1786829023.46 total 60.06 last 92.40)
 - open/close 6094 (time 1786829033.47 total 70.07 last 93.83)
 - open/close 6961 (time 1786829043.47 total 80.07 last 86.69)
 - open/close 7852 (time 1786829053.48 total 90.07 last 89.06)
 - open/close 8798 (time 1786829063.48 total 100.08 last 94.59)
 - open/close 9611 (time 1786829073.49 total 110.09 last 81.22)
total: 10000 open/close in 114.72 seconds: 87.17 ops/second
 - unlinked 0 (time 1786829081 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 937 (time 1786829166.22 total 10.00 last 93.68)
 - open/close 1834 (time 1786829176.22 total 20.00 last 89.70)
 - open/close 2747 (time 1786829186.23 total 30.01 last 91.21)
 - open/close 3642 (time 1786829196.24 total 40.02 last 89.44)
 - open/close 4495 (time 1786829206.24 total 50.02 last 85.26)
 - open/close 5414 (time 1786829216.25 total 60.03 last 91.86)
 - open/close 6295 (time 1786829226.25 total 70.03 last 88.06)
 - open/close 7163 (time 1786829236.27 total 80.05 last 86.62)
 - open/close 8049 (time 1786829246.28 total 90.06 last 88.53)
 - open/close 8998 (time 1786829256.28 total 100.07 last 94.84)
 - open/close 9888 (time 1786829266.29 total 110.07 last 88.94)
total: 10000 open/close in 111.52 seconds: 89.67 ops/second
 - unlinked 0 (time 1786829271 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg601-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg601-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg601-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1
Started lustre-OST0000
oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg601-server: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg601-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg601-client.virtnet:  -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg601-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg601-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.101@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.101@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.101@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.101@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.101@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg601-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1
Started lustre-OST0000
oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg601-server: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg601-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg601-client.virtnet:  -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg601-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg601-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server
stop mds service on oleg601-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg601-server

unloading modules via unload_modules_local on: 'oleg601-server'
oleg601-server: oleg601-server.virtnet: executing unload_modules_local
oleg601-server: 
modules unloaded.
pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2
pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2
pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2