== conf-sanity test 69: replace an OST with the same index ========================================================== 06:38:05 (1787395085)
start mds service on oleg132-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 542 (time 1787395173.96 total 10.00 last 54.18)
 - open/close 1186 (time 1787395183.98 total 20.02 last 64.32)
 - open/close 1848 (time 1787395193.99 total 30.03 last 66.14)
 - open/close 2535 (time 1787395203.99 total 40.03 last 68.67)
 - open/close 3247 (time 1787395213.99 total 50.03 last 71.17)
 - open/close 4002 (time 1787395224.00 total 60.04 last 75.49)
 - open/close 4837 (time 1787395234.01 total 70.05 last 83.41)
 - open/close 5625 (time 1787395244.01 total 80.05 last 78.78)
 - open/close 6347 (time 1787395254.01 total 90.05 last 72.17)
 - open/close 7167 (time 1787395264.03 total 100.07 last 81.88)
 - open/close 7913 (time 1787395274.03 total 110.07 last 74.60)
 - open/close 8694 (time 1787395284.03 total 120.07 last 78.05)
 - open/close 9527 (time 1787395294.04 total 130.08 last 83.24)
total: 10000 open/close in 135.97 seconds: 73.55 ops/second
 - unlinked 0 (time 1787395304 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 852 (time 1787395405.50 total 10.00 last 85.19)
 - open/close 1723 (time 1787395415.51 total 20.01 last 87.03)
 - open/close 2598 (time 1787395425.52 total 30.02 last 87.43)
 - open/close 3412 (time 1787395435.53 total 40.03 last 81.31)
 - open/close 4259 (time 1787395445.54 total 50.04 last 84.64)
 - open/close 4982 (time 1787395455.55 total 60.04 last 72.25)
 - open/close 5760 (time 1787395465.55 total 70.05 last 77.74)
 - open/close 6540 (time 1787395475.57 total 80.06 last 77.89)
 - open/close 7371 (time 1787395485.58 total 90.07 last 83.03)
 - open/close 8152 (time 1787395495.58 total 100.07 last 78.08)
 - open/close 8965 (time 1787395505.59 total 110.09 last 81.20)
 - open/close 9820 (time 1787395515.59 total 120.09 last 85.49)
total: 10000 open/close in 121.89 seconds: 82.04 ops/second
 - unlinked 0 (time 1787395521 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 791 (time 1787395616.85 total 10.01 last 79.01)
 - open/close 1646 (time 1787395626.86 total 20.02 last 85.44)
 - open/close 2596 (time 1787395636.87 total 30.03 last 94.92)
 - open/close 3517 (time 1787395646.87 total 40.03 last 92.08)
 - open/close 4420 (time 1787395656.87 total 50.03 last 90.27)
 - open/close 5199 (time 1787395666.88 total 60.04 last 77.85)
 - open/close 5996 (time 1787395676.89 total 70.05 last 79.63)
 - open/close 6679 (time 1787395686.91 total 80.07 last 68.17)
 - open/close 7341 (time 1787395696.91 total 90.07 last 66.15)
 - open/close 8135 (time 1787395706.92 total 100.08 last 79.39)
 - open/close 9010 (time 1787395716.92 total 110.08 last 87.42)
 - open/close 9822 (time 1787395726.94 total 120.10 last 81.09)
total: 10000 open/close in 122.84 seconds: 81.41 ops/second
 - unlinked 0 (time 1787395733 ; total 0 ; last 0)
total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second
 - open/close 944 (time 1787395817.59 total 10.01 last 94.31)
 - open/close 1874 (time 1787395827.60 total 20.01 last 92.98)
 - open/close 2672 (time 1787395837.60 total 30.02 last 79.73)
 - open/close 3442 (time 1787395847.61 total 40.03 last 76.93)
 - open/close 4315 (time 1787395857.62 total 50.04 last 87.21)
 - open/close 5335 (time 1787395867.63 total 60.04 last 101.98)
 - open/close 6164 (time 1787395877.63 total 70.05 last 82.83)
 - open/close 6941 (time 1787395887.64 total 80.05 last 77.68)
 - open/close 7753 (time 1787395897.65 total 90.06 last 81.12)
 - open/close 8549 (time 1787395907.66 total 100.07 last 79.51)
 - open/close 9373 (time 1787395917.66 total 110.08 last 82.35)
total: 10000 open/close in 119.31 seconds: 83.82 ops/second
 - unlinked 0 (time 1787395933 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 832 (time 1787396029.81 total 10.01 last 83.16)
 - open/close 1866 (time 1787396039.81 total 20.01 last 103.36)
 - open/close 2733 (time 1787396049.82 total 30.02 last 86.64)
 - open/close 3566 (time 1787396059.84 total 40.04 last 83.14)
 - open/close 4395 (time 1787396069.84 total 50.04 last 82.87)
 - open/close 5273 (time 1787396079.85 total 60.05 last 87.75)
 - open/close 6125 (time 1787396089.85 total 70.05 last 85.19)
 - open/close 7003 (time 1787396099.86 total 80.05 last 87.77)
 - open/close 7776 (time 1787396109.86 total 90.05 last 77.28)
 - open/close 8642 (time 1787396119.86 total 100.06 last 86.54)
 - open/close 9453 (time 1787396129.87 total 110.06 last 81.06)
total: 10000 open/close in 117.81 seconds: 84.88 ops/second
 - unlinked 0 (time 1787396141 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 833 (time 1787396231.31 total 10.00 last 83.27)
 - open/close 1652 (time 1787396241.31 total 20.01 last 81.88)
 - open/close 2446 (time 1787396251.32 total 30.02 last 79.31)
 - open/close 3210 (time 1787396261.32 total 40.02 last 76.40)
 - open/close 3902 (time 1787396271.32 total 50.02 last 69.20)
 - open/close 4771 (time 1787396281.33 total 60.02 last 86.87)
 - open/close 5666 (time 1787396291.33 total 70.02 last 89.50)
 - open/close 6404 (time 1787396301.33 total 80.03 last 73.77)
 - open/close 7184 (time 1787396311.34 total 90.04 last 77.92)
 - open/close 8069 (time 1787396321.34 total 100.04 last 88.49)
 - open/close 8963 (time 1787396331.35 total 110.04 last 89.39)
 - open/close 9718 (time 1787396341.35 total 120.05 last 75.43)
total: 10000 open/close in 123.97 seconds: 80.66 ops/second
 - unlinked 0 (time 1787396349 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 678 (time 1787396447.50 total 10.01 last 67.74)
 - open/close 1239 (time 1787396457.50 total 20.01 last 56.09)
 - open/close 1972 (time 1787396467.52 total 30.02 last 73.20)
 - open/close 2757 (time 1787396477.52 total 40.03 last 78.48)
 - open/close 3406 (time 1787396487.52 total 50.03 last 64.88)
 - open/close 3941 (time 1787396497.54 total 60.05 last 53.41)
 - open/close 4544 (time 1787396507.56 total 70.06 last 60.21)
 - open/close 5253 (time 1787396517.56 total 80.07 last 70.83)
 - open/close 5890 (time 1787396527.57 total 90.07 last 63.69)
 - open/close 6619 (time 1787396537.58 total 100.08 last 72.84)
 - open/close 7429 (time 1787396547.60 total 110.10 last 80.81)
 - open/close 8274 (time 1787396557.60 total 120.11 last 84.49)
 - open/close 9181 (time 1787396567.60 total 130.11 last 90.69)
 - open/close 9939 (time 1787396577.61 total 140.11 last 75.74)
total: 10000 open/close in 140.69 seconds: 71.08 ops/second
 - unlinked 0 (time 1787396581 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 800 (time 1787396677.90 total 10.01 last 79.93)
 - open/close 1407 (time 1787396687.90 total 20.02 last 60.66)
 - open/close 2207 (time 1787396697.93 total 30.04 last 79.82)
 - open/close 2962 (time 1787396707.93 total 40.04 last 75.46)
 - open/close 3652 (time 1787396717.96 total 50.07 last 68.83)
 - open/close 4440 (time 1787396727.97 total 60.08 last 78.74)
 - open/close 5311 (time 1787396737.97 total 70.09 last 87.02)
 - open/close 6156 (time 1787396747.98 total 80.09 last 84.49)
 - open/close 6905 (time 1787396757.98 total 90.09 last 74.90)
 - open/close 7603 (time 1787396767.99 total 100.10 last 69.73)
 - open/close 8247 (time 1787396777.99 total 110.10 last 64.38)
 - open/close 9051 (time 1787396788.00 total 120.11 last 80.30)
 - open/close 9859 (time 1787396798.00 total 130.12 last 80.78)
total: 10000 open/close in 131.99 seconds: 75.76 ops/second
 - unlinked 0 (time 1787396804 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 680 (time 1787396899.03 total 10.00 last 67.97)
 - open/close 1277 (time 1787396909.03 total 20.01 last 59.66)
 - open/close 2034 (time 1787396919.04 total 30.02 last 75.67)
 - open/close 2882 (time 1787396929.04 total 40.02 last 84.78)
 - open/close 3760 (time 1787396939.05 total 50.02 last 87.74)
 - open/close 4632 (time 1787396949.05 total 60.02 last 87.19)
 - open/close 5489 (time 1787396959.07 total 70.05 last 85.51)
 - open/close 6379 (time 1787396969.07 total 80.05 last 88.99)
 - open/close 7219 (time 1787396979.08 total 90.06 last 83.94)
 - open/close 7953 (time 1787396989.08 total 100.06 last 73.35)
 - open/close 8718 (time 1787396999.09 total 110.06 last 76.49)
 - open/close 9401 (time 1787397009.09 total 120.07 last 68.25)
total: 10000 open/close in 127.29 seconds: 78.56 ops/second
 - unlinked 0 (time 1787397020 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 732 (time 1787397128.46 total 10.02 last 73.08)
 - open/close 1505 (time 1787397138.47 total 20.02 last 77.26)
 - open/close 2290 (time 1787397148.48 total 30.03 last 78.45)
 - open/close 3095 (time 1787397158.48 total 40.04 last 80.43)
 - open/close 3770 (time 1787397168.52 total 50.07 last 67.27)
 - open/close 4496 (time 1787397178.52 total 60.08 last 72.56)
 - open/close 5185 (time 1787397188.53 total 70.09 last 68.84)
 - open/close 5957 (time 1787397198.55 total 80.10 last 77.11)
 - open/close 6649 (time 1787397208.55 total 90.10 last 69.18)
 - open/close 7377 (time 1787397218.56 total 100.11 last 72.73)
 - open/close 8106 (time 1787397228.57 total 110.12 last 72.84)
 - open/close 8842 (time 1787397238.58 total 120.13 last 73.52)
 - open/close 9562 (time 1787397248.60 total 130.15 last 71.87)
total: 10000 open/close in 135.93 seconds: 73.57 ops/second
 - unlinked 0 (time 1787397259 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.132@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.132@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server
stop mds service on oleg132-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server

unloading modules via unload_modules_local on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing unload_modules_local
oleg132-server: 
modules unloaded.
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2