== conf-sanity test 69: replace an OST with the same index ========================================================== 02:13:44 (1787552024)
start mds service on oleg147-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg147-server'
oleg147-server: oleg147-server.virtnet: executing load_modules_local
oleg147-server: Loading modules from /home/green/git/lustre-release/lustre
oleg147-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg147-server: detected 4 online CPUs by sysfs
oleg147-server: MODOPTS_LIBCFS=
oleg147-server: Force libcfs to create 2 CPU partitions
oleg147-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg147-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg147-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg147-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1
Started lustre-OST0000
oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg147-client.virtnet:  -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 621 (time 1787552123.05 total 10.01 last 62.02)
 - open/close 1280 (time 1787552133.05 total 20.02 last 65.88)
 - open/close 1936 (time 1787552143.07 total 30.03 last 65.51)
 - open/close 2621 (time 1787552153.07 total 40.03 last 68.50)
 - open/close 3376 (time 1787552163.07 total 50.04 last 75.45)
 - open/close 4006 (time 1787552173.08 total 60.04 last 62.98)
 - open/close 4672 (time 1787552183.09 total 70.05 last 66.53)
 - open/close 5320 (time 1787552193.09 total 80.05 last 64.78)
 - open/close 5984 (time 1787552203.09 total 90.05 last 66.40)
 - open/close 6589 (time 1787552213.10 total 100.06 last 60.45)
 - open/close 7411 (time 1787552223.11 total 110.07 last 82.14)
 - open/close 8090 (time 1787552233.11 total 120.07 last 67.87)
 - open/close 8708 (time 1787552243.11 total 130.08 last 61.80)
 - open/close 9341 (time 1787552253.12 total 140.09 last 63.24)
total: 10000 open/close in 148.90 seconds: 67.16 ops/second
 - unlinked 0 (time 1787552267 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 624 (time 1787552380.99 total 10.01 last 62.32)
 - open/close 1329 (time 1787552391.00 total 20.01 last 70.49)
 - open/close 1934 (time 1787552401.00 total 30.02 last 60.46)
 - open/close 2499 (time 1787552411.01 total 40.03 last 56.44)
 - open/close 2963 (time 1787552421.01 total 50.03 last 46.39)
 - open/close 3682 (time 1787552431.02 total 60.03 last 71.89)
 - open/close 4154 (time 1787552441.04 total 70.06 last 47.09)
 - open/close 4710 (time 1787552451.04 total 80.06 last 55.58)
 - open/close 5299 (time 1787552461.05 total 90.07 last 58.87)
 - open/close 5960 (time 1787552471.07 total 100.09 last 65.97)
 - open/close 6663 (time 1787552481.08 total 110.10 last 70.18)
 - open/close 7199 (time 1787552491.10 total 120.11 last 53.54)
 - open/close 7695 (time 1787552501.10 total 130.11 last 49.60)
 - open/close 8273 (time 1787552511.11 total 140.13 last 57.69)
 - open/close 8825 (time 1787552521.12 total 150.14 last 55.16)
 - open/close 9496 (time 1787552531.13 total 160.15 last 67.06)
 - open/close 9977 (time 1787552541.14 total 170.16 last 48.03)
total: 10000 open/close in 170.57 seconds: 58.63 ops/second
 - unlinked 0 (time 1787552547 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 669 (time 1787552656.98 total 10.00 last 66.87)
 - open/close 1443 (time 1787552666.98 total 20.01 last 77.34)
 - open/close 2173 (time 1787552677.00 total 30.03 last 72.87)
 - open/close 2881 (time 1787552687.01 total 40.04 last 70.74)
 - open/close 3595 (time 1787552697.01 total 50.04 last 71.40)
 - open/close 4340 (time 1787552707.02 total 60.04 last 74.45)
 - open/close 5074 (time 1787552717.03 total 70.05 last 73.33)
 - open/close 5685 (time 1787552727.03 total 80.06 last 61.09)
 - open/close 6292 (time 1787552737.04 total 90.07 last 60.63)
 - open/close 6946 (time 1787552747.04 total 100.07 last 65.37)
 - open/close 7619 (time 1787552757.06 total 110.09 last 67.19)
 - open/close 8157 (time 1787552767.06 total 120.09 last 53.77)
 - open/close 8841 (time 1787552777.07 total 130.10 last 68.38)
 - open/close 9657 (time 1787552787.08 total 140.11 last 81.49)
total: 10000 open/close in 143.89 seconds: 69.50 ops/second
 - unlinked 0 (time 1787552795 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 446 (time 1787552903.46 total 10.01 last 44.54)
 - open/close 985 (time 1787552913.46 total 20.01 last 53.89)
 - open/close 1435 (time 1787552923.46 total 30.02 last 44.99)
 - open/close 1930 (time 1787552933.47 total 40.03 last 49.45)
 - open/close 2655 (time 1787552943.48 total 50.03 last 72.47)
 - open/close 3343 (time 1787552953.48 total 60.04 last 68.77)
 - open/close 4141 (time 1787552963.51 total 70.06 last 79.60)
 - open/close 4820 (time 1787552973.52 total 80.08 last 67.79)
 - open/close 5495 (time 1787552983.53 total 90.09 last 67.43)
 - open/close 6109 (time 1787552993.54 total 100.09 last 61.40)
 - open/close 6662 (time 1787553003.54 total 110.09 last 55.28)
 - open/close 7271 (time 1787553013.54 total 120.10 last 60.87)
 - open/close 7972 (time 1787553023.55 total 130.10 last 70.07)
 - open/close 8572 (time 1787553033.56 total 140.11 last 59.93)
 - open/close 9217 (time 1787553043.56 total 150.12 last 64.48)
total: 10000 open/close in 160.08 seconds: 62.47 ops/second
 - unlinked 0 (time 1787553057 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 651 (time 1787553168.31 total 10.02 last 64.94)
 - open/close 1352 (time 1787553178.31 total 20.03 last 70.07)
 - open/close 2090 (time 1787553188.35 total 30.06 last 73.54)
 - open/close 2796 (time 1787553198.35 total 40.06 last 70.59)
 - open/close 3501 (time 1787553208.35 total 50.07 last 70.47)
 - open/close 4022 (time 1787553218.36 total 60.08 last 52.06)
 - open/close 4458 (time 1787553228.37 total 70.08 last 43.57)
 - open/close 4946 (time 1787553238.38 total 80.10 last 48.74)
 - open/close 5629 (time 1787553248.39 total 90.10 last 68.27)
 - open/close 6336 (time 1787553258.39 total 100.11 last 70.65)
 - open/close 6794 (time 1787553268.40 total 110.12 last 45.76)
 - open/close 7418 (time 1787553278.42 total 120.14 last 62.28)
 - open/close 7875 (time 1787553288.43 total 130.14 last 45.69)
 - open/close 8465 (time 1787553298.43 total 140.14 last 58.97)
 - open/close 9118 (time 1787553308.43 total 150.15 last 65.29)
 - open/close 9636 (time 1787553318.46 total 160.17 last 51.67)
total: 10000 open/close in 165.33 seconds: 60.48 ops/second
 - unlinked 0 (time 1787553329 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 618 (time 1787553442.03 total 10.00 last 61.78)
 - open/close 1343 (time 1787553452.05 total 20.02 last 72.40)
 - open/close 2087 (time 1787553462.05 total 30.02 last 74.36)
 - open/close 2694 (time 1787553472.06 total 40.03 last 60.63)
 - open/close 3376 (time 1787553482.07 total 50.04 last 68.14)
 - open/close 4093 (time 1787553492.09 total 60.06 last 71.61)
 - open/close 4685 (time 1787553502.09 total 70.06 last 59.15)
 - open/close 5275 (time 1787553512.11 total 80.08 last 58.93)
 - open/close 5806 (time 1787553522.11 total 90.08 last 53.09)
 - open/close 6455 (time 1787553532.12 total 100.09 last 64.82)
 - open/close 7115 (time 1787553542.12 total 110.09 last 65.97)
 - open/close 7576 (time 1787553552.14 total 120.11 last 46.02)
 - open/close 8167 (time 1787553562.15 total 130.12 last 59.06)
 - open/close 8794 (time 1787553572.15 total 140.12 last 62.69)
 - open/close 9522 (time 1787553582.15 total 150.12 last 72.79)
total: 10000 open/close in 158.16 seconds: 63.23 ops/second
 - unlinked 0 (time 1787553595 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 540 (time 1787553711.30 total 10.03 last 53.83)
 - open/close 1152 (time 1787553721.32 total 20.04 last 61.12)
 - open/close 1817 (time 1787553731.32 total 30.05 last 66.49)
 - open/close 2476 (time 1787553741.32 total 40.05 last 65.86)
 - open/close 3035 (time 1787553751.33 total 50.05 last 55.89)
 - open/close 3523 (time 1787553761.33 total 60.06 last 48.77)
 - open/close 3967 (time 1787553771.34 total 70.07 last 44.36)
 - open/close 4516 (time 1787553781.37 total 80.09 last 54.77)
 - open/close 5140 (time 1787553791.37 total 90.10 last 62.37)
 - open/close 5837 (time 1787553801.40 total 100.13 last 69.51)
 - open/close 6372 (time 1787553811.40 total 110.13 last 53.49)
 - open/close 7018 (time 1787553821.41 total 120.13 last 64.57)
 - open/close 7708 (time 1787553831.41 total 130.14 last 68.98)
 - open/close 8521 (time 1787553841.42 total 140.15 last 81.21)
 - open/close 9103 (time 1787553851.42 total 150.15 last 58.20)
 - open/close 9526 (time 1787553861.42 total 160.15 last 42.29)
total: 10000 open/close in 169.75 seconds: 58.91 ops/second
 - unlinked 0 (time 1787553876 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 720 (time 1787553992.25 total 10.01 last 71.95)
 - open/close 1410 (time 1787554002.26 total 20.02 last 68.93)
 - open/close 2180 (time 1787554012.26 total 30.02 last 76.97)
 - open/close 2949 (time 1787554022.29 total 40.05 last 76.68)
 - open/close 3684 (time 1787554032.29 total 50.05 last 73.49)
 - open/close 4286 (time 1787554042.30 total 60.06 last 60.14)
 - open/close 4995 (time 1787554052.31 total 70.07 last 70.87)
 - open/close 5637 (time 1787554062.32 total 80.08 last 64.13)
 - open/close 6179 (time 1787554072.32 total 90.08 last 54.17)
 - open/close 6790 (time 1787554082.32 total 100.08 last 61.09)
 - open/close 7328 (time 1787554092.34 total 110.10 last 53.73)
 - open/close 7781 (time 1787554102.35 total 120.11 last 45.23)
 - open/close 8171 (time 1787554112.37 total 130.13 last 38.94)
 - open/close 8747 (time 1787554122.38 total 140.14 last 57.50)
 - open/close 9157 (time 1787554132.39 total 150.15 last 40.98)
 - open/close 9726 (time 1787554142.39 total 160.15 last 56.87)
total: 10000 open/close in 164.39 seconds: 60.83 ops/second
 - unlinked 0 (time 1787554152 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 622 (time 1787554275.27 total 10.00 last 62.17)
 - open/close 1289 (time 1787554285.29 total 20.02 last 66.59)
 - open/close 1979 (time 1787554295.29 total 30.03 last 68.95)
 - open/close 2634 (time 1787554305.31 total 40.04 last 65.41)
 - open/close 3218 (time 1787554315.31 total 50.04 last 58.39)
 - open/close 3770 (time 1787554325.33 total 60.06 last 55.11)
 - open/close 4407 (time 1787554335.34 total 70.07 last 63.63)
 - open/close 5005 (time 1787554345.35 total 80.08 last 59.75)
 - open/close 5667 (time 1787554355.35 total 90.09 last 66.17)
 - open/close 6316 (time 1787554365.36 total 100.10 last 64.82)
 - open/close 6845 (time 1787554375.36 total 110.10 last 52.89)
 - open/close 7404 (time 1787554385.38 total 120.11 last 55.83)
 - open/close 8085 (time 1787554395.38 total 130.12 last 68.06)
 - open/close 8742 (time 1787554405.39 total 140.13 last 65.63)
 - open/close 9397 (time 1787554415.40 total 150.14 last 65.42)
total: 10000 open/close in 158.73 seconds: 63.00 ops/second
 - unlinked 0 (time 1787554428 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 745 (time 1787554537.78 total 10.03 last 74.28)
 - open/close 1436 (time 1787554547.79 total 20.04 last 69.03)
 - open/close 2114 (time 1787554557.80 total 30.04 last 67.78)
 - open/close 2835 (time 1787554567.80 total 40.05 last 72.07)
 - open/close 3570 (time 1787554577.81 total 50.05 last 73.44)
 - open/close 4200 (time 1787554587.81 total 60.06 last 62.98)
 - open/close 4831 (time 1787554597.81 total 70.06 last 63.09)
 - open/close 5727 (time 1787554607.82 total 80.06 last 89.58)
 - open/close 6585 (time 1787554617.82 total 90.07 last 85.78)
 - open/close 7416 (time 1787554627.83 total 100.07 last 83.03)
 - open/close 8209 (time 1787554637.83 total 110.08 last 79.25)
 - open/close 8977 (time 1787554647.84 total 120.09 last 76.76)
 - open/close 9790 (time 1787554657.84 total 130.09 last 81.30)
total: 10000 open/close in 132.22 seconds: 75.63 ops/second
 - unlinked 0 (time 1787554664 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg147-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg147-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.147@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.147@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg147-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1
Started lustre-OST0000
oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg147-server: oleg147-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg147-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg147-client.virtnet:  -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre
On OST0, 17766 used inodes rc=0
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg147-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg147-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.147@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.147@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.147@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.147@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.147@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg147-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1
Started lustre-OST0000
oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg147-server: oleg147-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg147-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg147-client.virtnet:  -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg147-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg147-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server
stop mds service on oleg147-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server

unloading modules via unload_modules_local on: 'oleg147-server'
oleg147-server: oleg147-server.virtnet: executing unload_modules_local
oleg147-server: 
modules unloaded.
pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2
pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2
pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2