== conf-sanity test 69: replace an OST with the same index ========================================================== 03:33:18 (1780990398)
start mds service on oleg121-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg121-server'
oleg121-server: oleg121-server.virtnet: executing load_modules_local
oleg121-server: Loading modules from /home/green/git/lustre-release/lustre
oleg121-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg121-server: detected 4 online CPUs by sysfs
oleg121-server: MODOPTS_LIBCFS=
oleg121-server: Force libcfs to create 2 CPU partitions
oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg121-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 741 (time 1780990511.70 total 10.00 last 74.07)
 - open/close 1313 (time 1780990521.70 total 20.01 last 57.16)
 - open/close 1945 (time 1780990531.71 total 30.02 last 63.14)
 - open/close 2749 (time 1780990541.72 total 40.03 last 80.33)
 - open/close 3468 (time 1780990551.73 total 50.03 last 71.87)
 - open/close 4190 (time 1780990561.74 total 60.05 last 72.09)
 - open/close 4959 (time 1780990571.75 total 70.06 last 76.84)
 - open/close 5562 (time 1780990581.76 total 80.06 last 60.27)
 - open/close 6266 (time 1780990591.77 total 90.08 last 70.29)
 - open/close 7025 (time 1780990601.77 total 100.08 last 75.88)
 - open/close 7774 (time 1780990611.78 total 110.08 last 74.87)
 - open/close 8400 (time 1780990621.80 total 120.11 last 62.46)
 - open/close 8848 (time 1780990631.80 total 130.11 last 44.78)
 - open/close 9269 (time 1780990641.82 total 140.13 last 42.04)
 - open/close 9919 (time 1780990651.82 total 150.13 last 64.98)
total: 10000 open/close in 151.43 seconds: 66.04 ops/second
 - unlinked 0 (time 1780990658 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 674 (time 1780990766.51 total 10.01 last 67.34)
 - open/close 1463 (time 1780990776.52 total 20.01 last 78.86)
 - open/close 2139 (time 1780990786.55 total 30.04 last 67.40)
 - open/close 2701 (time 1780990796.56 total 40.06 last 56.14)
 - open/close 3276 (time 1780990806.57 total 50.07 last 57.43)
 - open/close 4041 (time 1780990816.58 total 60.08 last 76.44)
 - open/close 4777 (time 1780990826.59 total 70.09 last 73.52)
 - open/close 5539 (time 1780990836.60 total 80.10 last 76.12)
 - open/close 6301 (time 1780990846.60 total 90.10 last 76.16)
 - open/close 7083 (time 1780990856.60 total 100.10 last 78.19)
 - open/close 7813 (time 1780990866.61 total 110.11 last 72.94)
 - open/close 8459 (time 1780990876.65 total 120.15 last 64.35)
 - open/close 9154 (time 1780990886.65 total 130.15 last 69.48)
 - open/close 9878 (time 1780990896.66 total 140.16 last 72.36)
total: 10000 open/close in 141.84 seconds: 70.50 ops/second
 - unlinked 0 (time 1780990904 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 837 (time 1780991003.78 total 10.01 last 83.62)
 - open/close 1598 (time 1780991013.80 total 20.03 last 75.94)
 - open/close 2406 (time 1780991023.81 total 30.04 last 80.71)
 - open/close 3198 (time 1780991033.82 total 40.06 last 79.09)
 - open/close 3906 (time 1780991043.84 total 50.07 last 70.71)
 - open/close 4532 (time 1780991053.84 total 60.07 last 62.57)
 - open/close 5239 (time 1780991063.84 total 70.08 last 70.69)
 - open/close 5899 (time 1780991073.85 total 80.09 last 65.93)
 - open/close 6523 (time 1780991083.87 total 90.10 last 62.31)
 - open/close 7193 (time 1780991093.87 total 100.10 last 66.97)
 - open/close 7866 (time 1780991103.87 total 110.10 last 67.29)
 - open/close 8538 (time 1780991113.89 total 120.12 last 67.10)
 - open/close 9424 (time 1780991123.89 total 130.12 last 88.59)
total: 10000 open/close in 137.62 seconds: 72.66 ops/second
 - unlinked 0 (time 1780991135 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 815 (time 1780991226.24 total 10.00 last 81.49)
 - open/close 1470 (time 1780991236.25 total 20.01 last 65.44)
 - open/close 2202 (time 1780991246.27 total 30.03 last 73.05)
 - open/close 2955 (time 1780991256.28 total 40.04 last 75.23)
 - open/close 3787 (time 1780991266.29 total 50.04 last 83.16)
 - open/close 4553 (time 1780991276.30 total 60.05 last 76.53)
 - open/close 5312 (time 1780991286.33 total 70.09 last 75.64)
 - open/close 6030 (time 1780991296.34 total 80.10 last 71.71)
 - open/close 6816 (time 1780991306.37 total 90.13 last 78.39)
 - open/close 7641 (time 1780991316.38 total 100.13 last 82.44)
 - open/close 8288 (time 1780991326.39 total 110.15 last 64.60)
 - open/close 9179 (time 1780991336.39 total 120.15 last 89.09)
total: 10000 open/close in 129.26 seconds: 77.36 ops/second
 - unlinked 0 (time 1780991350 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 771 (time 1780991450.87 total 10.00 last 77.07)
 - open/close 1531 (time 1780991460.89 total 20.03 last 75.80)
 - open/close 2219 (time 1780991470.90 total 30.04 last 68.75)
 - open/close 2951 (time 1780991480.90 total 40.04 last 73.17)
 - open/close 3674 (time 1780991490.91 total 50.05 last 72.25)
 - open/close 4388 (time 1780991500.91 total 60.05 last 71.40)
 - open/close 5136 (time 1780991510.93 total 70.07 last 74.63)
 - open/close 5811 (time 1780991520.95 total 80.09 last 67.41)
 - open/close 6325 (time 1780991530.98 total 90.11 last 51.25)
 - open/close 6878 (time 1780991540.98 total 100.12 last 55.29)
 - open/close 7707 (time 1780991550.98 total 110.12 last 82.90)
 - open/close 8612 (time 1780991560.98 total 120.12 last 90.48)
 - open/close 9629 (time 1780991570.99 total 130.13 last 101.61)
total: 10000 open/close in 134.85 seconds: 74.16 ops/second
 - unlinked 0 (time 1780991579 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 892 (time 1780991667.21 total 10.03 last 88.90)
 - open/close 1595 (time 1780991677.21 total 20.03 last 70.30)
 - open/close 2316 (time 1780991687.21 total 30.04 last 72.07)
 - open/close 3256 (time 1780991697.23 total 40.06 last 93.78)
 - open/close 4001 (time 1780991707.25 total 50.07 last 74.40)
 - open/close 4725 (time 1780991717.25 total 60.08 last 72.38)
 - open/close 5539 (time 1780991727.26 total 70.09 last 81.31)
 - open/close 6346 (time 1780991737.27 total 80.09 last 80.64)
 - open/close 7055 (time 1780991747.27 total 90.10 last 70.87)
 - open/close 7550 (time 1780991757.30 total 100.12 last 49.37)
 - open/close 8280 (time 1780991767.30 total 110.13 last 72.97)
 - open/close 9131 (time 1780991777.32 total 120.14 last 84.97)
 - open/close 9914 (time 1780991787.32 total 130.15 last 78.27)
total: 10000 open/close in 131.32 seconds: 76.15 ops/second
 - unlinked 0 (time 1780991793 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 714 (time 1780991884.13 total 10.01 last 71.36)
 - open/close 1495 (time 1780991894.14 total 20.01 last 78.03)
 - open/close 2254 (time 1780991904.14 total 30.02 last 75.89)
 - open/close 3129 (time 1780991914.15 total 40.02 last 87.47)
 - open/close 4097 (time 1780991924.15 total 50.02 last 96.80)
 - open/close 4928 (time 1780991934.15 total 60.03 last 83.04)
 - open/close 5737 (time 1780991944.16 total 70.03 last 80.84)
 - open/close 6547 (time 1780991954.17 total 80.04 last 80.95)
 - open/close 7195 (time 1780991964.18 total 90.05 last 64.72)
 - open/close 7829 (time 1780991974.19 total 100.06 last 63.36)
 - open/close 8638 (time 1780991984.19 total 110.06 last 80.90)
 - open/close 9469 (time 1780991994.19 total 120.06 last 83.08)
total: 10000 open/close in 126.24 seconds: 79.21 ops/second
 - unlinked 0 (time 1780992005 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 797 (time 1780992093.79 total 10.00 last 79.66)
 - open/close 1659 (time 1780992103.80 total 20.01 last 86.13)
 - open/close 2489 (time 1780992113.80 total 30.02 last 82.97)
 - open/close 3178 (time 1780992123.81 total 40.02 last 68.87)
 - open/close 3811 (time 1780992133.81 total 50.03 last 63.27)
 - open/close 4673 (time 1780992143.82 total 60.03 last 86.16)
 - open/close 5580 (time 1780992153.83 total 70.04 last 90.61)
 - open/close 6315 (time 1780992163.83 total 80.04 last 73.46)
 - open/close 7154 (time 1780992173.83 total 90.05 last 83.88)
 - open/close 7926 (time 1780992183.86 total 100.07 last 77.02)
 - open/close 8445 (time 1780992193.87 total 110.09 last 51.82)
 - open/close 9003 (time 1780992203.87 total 120.09 last 55.80)
 - open/close 9666 (time 1780992213.88 total 130.09 last 66.24)
total: 10000 open/close in 134.83 seconds: 74.17 ops/second
 - unlinked 0 (time 1780992223 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 787 (time 1780992313.52 total 10.00 last 78.68)
 - open/close 1540 (time 1780992323.52 total 20.00 last 75.29)
 - open/close 2276 (time 1780992333.52 total 30.01 last 73.55)
 - open/close 2988 (time 1780992343.56 total 40.05 last 70.94)
 - open/close 3633 (time 1780992353.58 total 50.06 last 64.39)
 - open/close 4183 (time 1780992363.60 total 60.09 last 54.87)
 - open/close 4946 (time 1780992373.60 total 70.09 last 76.27)
 - open/close 5731 (time 1780992383.61 total 80.09 last 78.50)
 - open/close 6577 (time 1780992393.62 total 90.10 last 84.51)
 - open/close 7303 (time 1780992403.63 total 100.12 last 72.50)
 - open/close 8022 (time 1780992413.64 total 110.13 last 71.82)
 - open/close 8576 (time 1780992423.65 total 120.14 last 55.34)
 - open/close 9209 (time 1780992433.66 total 130.15 last 63.24)
total: 10000 open/close in 140.00 seconds: 71.43 ops/second
 - unlinked 0 (time 1780992449 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 663 (time 1780992549.25 total 10.00 last 66.29)
 - open/close 1404 (time 1780992559.28 total 20.03 last 73.93)
 - open/close 1965 (time 1780992569.29 total 30.03 last 56.06)
 - open/close 2720 (time 1780992579.30 total 40.05 last 75.38)
 - open/close 3535 (time 1780992589.31 total 50.05 last 81.46)
 - open/close 4061 (time 1780992599.31 total 60.06 last 52.58)
 - open/close 4572 (time 1780992609.34 total 70.09 last 50.94)
 - open/close 5138 (time 1780992619.35 total 80.10 last 56.55)
 - open/close 5660 (time 1780992629.36 total 90.11 last 52.13)
 - open/close 6291 (time 1780992639.36 total 100.11 last 63.10)
 - open/close 7113 (time 1780992649.38 total 110.12 last 82.10)
 - open/close 7910 (time 1780992659.38 total 120.12 last 79.69)
 - open/close 8748 (time 1780992669.39 total 130.14 last 83.68)
 - open/close 9472 (time 1780992679.39 total 140.14 last 72.40)
total: 10000 open/close in 149.30 seconds: 66.98 ops/second
 - unlinked 0 (time 1780992694 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
On OST0, 20519 used inodes rc=0
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.121@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.121@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.121@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg121-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1
Started lustre-OST0000
oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg121-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg121-client.virtnet:  -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg121-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg121-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server
stop mds service on oleg121-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server
stop mds service on oleg121-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg121-server

unloading modules via unload_modules_local on: 'oleg121-server'
oleg121-server: oleg121-server.virtnet: executing unload_modules_local
oleg121-server: 
modules unloaded.
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2
pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2