== conf-sanity test 69: replace an OST with the same index ========================================================== 01:35:06 (1788154506)
start mds service on oleg339-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing load_modules_local
oleg339-server: Loading modules from /home/green/git/lustre-release/lustre
oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg339-server: detected 4 online CPUs by sysfs
oleg339-server: MODOPTS_LIBCFS=
oleg339-server: Force libcfs to create 2 CPU partitions
oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg339-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-OST0000
oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg339-client.virtnet:  -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 610 (time 1788154607.50 total 10.02 last 60.87)
 - open/close 1178 (time 1788154617.51 total 20.03 last 56.75)
 - open/close 1799 (time 1788154627.51 total 30.03 last 62.07)
 - open/close 2497 (time 1788154637.51 total 40.03 last 69.79)
 - open/close 3174 (time 1788154647.53 total 50.05 last 67.62)
 - open/close 3796 (time 1788154657.54 total 60.06 last 62.10)
 - open/close 4447 (time 1788154667.56 total 70.08 last 65.00)
 - open/close 5088 (time 1788154677.57 total 80.09 last 64.00)
 - open/close 5712 (time 1788154687.58 total 90.10 last 62.34)
 - open/close 6432 (time 1788154697.58 total 100.10 last 71.99)
 - open/close 7082 (time 1788154707.59 total 110.11 last 64.95)
 - open/close 7727 (time 1788154717.60 total 120.12 last 64.45)
 - open/close 8320 (time 1788154727.61 total 130.13 last 59.26)
 - open/close 9032 (time 1788154737.63 total 140.15 last 71.06)
 - open/close 9693 (time 1788154747.64 total 150.16 last 66.05)
total: 10000 open/close in 155.11 seconds: 64.47 ops/second
 - unlinked 0 (time 1788154756 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 615 (time 1788154870.64 total 10.02 last 61.36)
 - open/close 1236 (time 1788154880.65 total 20.04 last 62.02)
 - open/close 1915 (time 1788154890.65 total 30.04 last 67.90)
 - open/close 2568 (time 1788154900.66 total 40.05 last 65.24)
 - open/close 3182 (time 1788154910.66 total 50.05 last 61.38)
 - open/close 3806 (time 1788154920.67 total 60.06 last 62.34)
 - open/close 4415 (time 1788154930.68 total 70.06 last 60.87)
 - open/close 4982 (time 1788154940.68 total 80.07 last 56.66)
 - open/close 5535 (time 1788154950.71 total 90.09 last 55.17)
 - open/close 6113 (time 1788154960.71 total 100.09 last 57.80)
 - open/close 6755 (time 1788154970.71 total 110.10 last 64.16)
 - open/close 7349 (time 1788154980.72 total 120.11 last 59.37)
 - open/close 7949 (time 1788154990.74 total 130.13 last 59.86)
 - open/close 8603 (time 1788155000.74 total 140.13 last 65.39)
 - open/close 9155 (time 1788155010.77 total 150.16 last 55.05)
 - open/close 9701 (time 1788155020.77 total 160.16 last 54.60)
total: 10000 open/close in 166.72 seconds: 59.98 ops/second
 - unlinked 0 (time 1788155032 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 674 (time 1788155144.51 total 10.01 last 67.33)
 - open/close 1301 (time 1788155154.51 total 20.01 last 62.69)
 - open/close 1933 (time 1788155164.51 total 30.02 last 63.17)
 - open/close 2563 (time 1788155174.51 total 40.02 last 63.00)
 - open/close 3174 (time 1788155184.52 total 50.03 last 61.05)
 - open/close 3782 (time 1788155194.52 total 60.03 last 60.79)
 - open/close 4394 (time 1788155204.53 total 70.04 last 61.14)
 - open/close 5048 (time 1788155214.56 total 80.07 last 65.20)
 - open/close 5635 (time 1788155224.56 total 90.07 last 58.69)
 - open/close 6170 (time 1788155234.57 total 100.07 last 53.50)
 - open/close 6759 (time 1788155244.57 total 110.07 last 58.89)
 - open/close 7378 (time 1788155254.57 total 120.07 last 61.89)
 - open/close 7980 (time 1788155264.58 total 130.08 last 60.14)
 - open/close 8563 (time 1788155274.60 total 140.10 last 58.19)
 - open/close 9157 (time 1788155284.61 total 150.11 last 59.34)
 - open/close 9731 (time 1788155294.63 total 160.13 last 57.30)
total: 10000 open/close in 164.45 seconds: 60.81 ops/second
 - unlinked 0 (time 1788155303 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 684 (time 1788155419.05 total 10.01 last 68.36)
 - open/close 1407 (time 1788155429.05 total 20.01 last 72.28)
 - open/close 2131 (time 1788155439.06 total 30.01 last 72.37)
 - open/close 2842 (time 1788155449.06 total 40.01 last 71.10)
 - open/close 3499 (time 1788155459.06 total 50.02 last 65.67)
 - open/close 4166 (time 1788155469.07 total 60.02 last 66.67)
 - open/close 4853 (time 1788155479.08 total 70.03 last 68.64)
 - open/close 5506 (time 1788155489.10 total 80.05 last 65.15)
 - open/close 6161 (time 1788155499.14 total 90.09 last 65.25)
 - open/close 6820 (time 1788155509.15 total 100.11 last 65.79)
 - open/close 7486 (time 1788155519.15 total 110.11 last 66.60)
 - open/close 8113 (time 1788155529.17 total 120.13 last 62.58)
 - open/close 8803 (time 1788155539.18 total 130.14 last 68.93)
 - open/close 9447 (time 1788155549.19 total 140.14 last 64.38)
total: 10000 open/close in 148.66 seconds: 67.27 ops/second
 - unlinked 0 (time 1788155562 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 674 (time 1788155685.64 total 10.01 last 67.34)
 - open/close 1348 (time 1788155695.67 total 20.04 last 67.18)
 - open/close 1986 (time 1788155705.70 total 30.07 last 63.59)
 - open/close 2632 (time 1788155715.71 total 40.08 last 64.56)
 - open/close 3322 (time 1788155725.72 total 50.09 last 68.92)
 - open/close 3977 (time 1788155735.73 total 60.10 last 65.45)
 - open/close 4668 (time 1788155745.73 total 70.10 last 69.07)
 - open/close 5337 (time 1788155755.73 total 80.10 last 66.89)
 - open/close 5987 (time 1788155765.74 total 90.12 last 64.93)
 - open/close 6652 (time 1788155775.74 total 100.12 last 66.50)
 - open/close 7299 (time 1788155785.75 total 110.12 last 64.66)
 - open/close 7952 (time 1788155795.75 total 120.12 last 65.29)
 - open/close 8606 (time 1788155805.77 total 130.14 last 65.31)
 - open/close 9252 (time 1788155815.77 total 140.14 last 64.59)
 - open/close 9879 (time 1788155825.79 total 150.16 last 62.59)
total: 10000 open/close in 152.01 seconds: 65.79 ops/second
 - unlinked 0 (time 1788155831 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 597 (time 1788155940.05 total 10.01 last 59.63)
 - open/close 1285 (time 1788155950.06 total 20.02 last 68.75)
 - open/close 1927 (time 1788155960.09 total 30.04 last 64.04)
 - open/close 2544 (time 1788155970.09 total 40.05 last 61.65)
 - open/close 3227 (time 1788155980.10 total 50.06 last 68.23)
 - open/close 3857 (time 1788155990.11 total 60.06 last 62.99)
 - open/close 4495 (time 1788156000.11 total 70.07 last 63.75)
 - open/close 5126 (time 1788156010.13 total 80.08 last 63.02)
 - open/close 5753 (time 1788156020.13 total 90.09 last 62.67)
 - open/close 6368 (time 1788156030.13 total 100.09 last 61.49)
 - open/close 6944 (time 1788156040.14 total 110.10 last 57.58)
 - open/close 7591 (time 1788156050.14 total 120.10 last 64.67)
 - open/close 8187 (time 1788156060.14 total 130.10 last 59.60)
 - open/close 8784 (time 1788156070.16 total 140.11 last 59.62)
 - open/close 9362 (time 1788156080.18 total 150.14 last 57.65)
 - open/close 9966 (time 1788156090.21 total 160.17 last 60.25)
total: 10000 open/close in 160.62 seconds: 62.26 ops/second
 - unlinked 0 (time 1788156095 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 655 (time 1788156208.68 total 10.00 last 65.49)
 - open/close 1324 (time 1788156218.70 total 20.02 last 66.79)
 - open/close 2034 (time 1788156228.71 total 30.03 last 70.93)
 - open/close 2718 (time 1788156238.72 total 40.04 last 68.34)
 - open/close 3420 (time 1788156248.73 total 50.04 last 70.14)
 - open/close 4051 (time 1788156258.73 total 60.05 last 63.06)
 - open/close 4693 (time 1788156268.73 total 70.05 last 64.19)
 - open/close 5313 (time 1788156278.76 total 80.07 last 61.87)
 - open/close 5931 (time 1788156288.77 total 90.09 last 61.72)
 - open/close 6572 (time 1788156298.77 total 100.09 last 64.10)
 - open/close 7182 (time 1788156308.79 total 110.11 last 60.84)
 - open/close 7805 (time 1788156318.80 total 120.12 last 62.25)
 - open/close 8383 (time 1788156328.80 total 130.12 last 57.78)
 - open/close 9008 (time 1788156338.82 total 140.14 last 62.42)
 - open/close 9647 (time 1788156348.82 total 150.14 last 63.87)
total: 10000 open/close in 155.94 seconds: 64.13 ops/second
 - unlinked 0 (time 1788156359 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 564 (time 1788156472.01 total 10.00 last 56.38)
 - open/close 1174 (time 1788156482.05 total 20.04 last 60.75)
 - open/close 1801 (time 1788156492.07 total 30.06 last 62.59)
 - open/close 2417 (time 1788156502.08 total 40.07 last 61.58)
 - open/close 3104 (time 1788156512.09 total 50.08 last 68.60)
 - open/close 3799 (time 1788156522.10 total 60.09 last 69.46)
 - open/close 4510 (time 1788156532.10 total 70.09 last 71.10)
 - open/close 5196 (time 1788156542.11 total 80.10 last 68.50)
 - open/close 5804 (time 1788156552.11 total 90.10 last 60.80)
 - open/close 6507 (time 1788156562.12 total 100.11 last 70.23)
 - open/close 7168 (time 1788156572.13 total 110.12 last 66.05)
 - open/close 7745 (time 1788156582.14 total 120.13 last 57.61)
 - open/close 8304 (time 1788156592.16 total 130.15 last 55.82)
 - open/close 8951 (time 1788156602.17 total 140.16 last 64.60)
 - open/close 9613 (time 1788156612.18 total 150.17 last 66.14)
total: 10000 open/close in 156.49 seconds: 63.90 ops/second
 - unlinked 0 (time 1788156623 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 630 (time 1788156731.47 total 10.01 last 62.94)
 - open/close 1251 (time 1788156741.47 total 20.01 last 62.09)
 - open/close 1892 (time 1788156751.48 total 30.02 last 64.05)
 - open/close 2525 (time 1788156761.48 total 40.02 last 63.28)
 - open/close 3180 (time 1788156771.50 total 50.04 last 65.41)
 - open/close 3817 (time 1788156781.53 total 60.07 last 63.52)
 - open/close 4391 (time 1788156791.54 total 70.08 last 57.34)
 - open/close 4991 (time 1788156801.55 total 80.09 last 59.92)
 - open/close 5609 (time 1788156811.55 total 90.09 last 61.77)
 - open/close 6261 (time 1788156821.56 total 100.10 last 65.17)
 - open/close 6847 (time 1788156831.57 total 110.11 last 58.55)
 - open/close 7431 (time 1788156841.58 total 120.12 last 58.34)
 - open/close 7963 (time 1788156851.59 total 130.13 last 53.12)
 - open/close 8631 (time 1788156861.60 total 140.14 last 66.77)
 - open/close 9231 (time 1788156871.61 total 150.15 last 59.94)
 - open/close 9844 (time 1788156881.61 total 160.15 last 61.25)
total: 10000 open/close in 162.60 seconds: 61.50 ops/second
 - unlinked 0 (time 1788156889 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 547 (time 1788157003.88 total 10.01 last 54.66)
 - open/close 916 (time 1788157013.89 total 20.01 last 36.87)
 - open/close 1295 (time 1788157023.91 total 30.03 last 37.84)
 - open/close 1766 (time 1788157033.93 total 40.05 last 46.99)
 - open/close 2286 (time 1788157043.94 total 50.06 last 51.94)
 - open/close 2887 (time 1788157053.95 total 60.07 last 60.06)
 - open/close 3579 (time 1788157063.95 total 70.07 last 69.18)
 - open/close 4191 (time 1788157073.96 total 80.09 last 61.12)
 - open/close 4708 (time 1788157083.97 total 90.09 last 51.67)
 - open/close 5267 (time 1788157093.97 total 100.10 last 55.87)
 - open/close 5943 (time 1788157103.98 total 110.11 last 67.53)
 - open/close 6423 (time 1788157113.98 total 120.11 last 48.00)
 - open/close 7005 (time 1788157124.00 total 130.12 last 58.12)
 - open/close 7628 (time 1788157134.01 total 140.13 last 62.26)
 - open/close 8232 (time 1788157144.01 total 150.13 last 60.40)
 - open/close 8821 (time 1788157154.01 total 160.14 last 58.85)
 - open/close 9364 (time 1788157164.02 total 170.14 last 54.29)
 - open/close 9958 (time 1788157174.03 total 180.15 last 59.32)
total: 10000 open/close in 180.83 seconds: 55.30 ops/second
 - unlinked 0 (time 1788157179 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg339-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg339-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-OST0000
oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg339-client.virtnet:  -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg339-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.139@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.139@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.139@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg339-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-OST0000
oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg339-client.virtnet:  -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg339-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server
stop mds service on oleg339-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server

unloading modules via unload_modules_local on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing unload_modules_local
oleg339-server: 
modules unloaded.
pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2
pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2