== conf-sanity test 69: replace an OST with the same index ========================================================== 14:24:53 (1780338293)
start mds service on oleg451-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg451-server'
oleg451-server: oleg451-server.virtnet: executing load_modules_local
oleg451-server: Loading modules from /home/green/git/lustre-release/lustre
oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg451-server: detected 4 online CPUs by sysfs
oleg451-server: MODOPTS_LIBCFS=
oleg451-server: Force libcfs to create 2 CPU partitions
oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg451-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1
Started lustre-OST0000
oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg451-client.virtnet:  -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 548 (time 1780338380.24 total 10.02 last 54.71)
 - open/close 1049 (time 1780338390.27 total 20.04 last 49.97)
 - open/close 1697 (time 1780338400.29 total 30.06 last 64.70)
 - open/close 2305 (time 1780338410.31 total 40.09 last 60.64)
 - open/close 2934 (time 1780338420.33 total 50.10 last 62.82)
 - open/close 3520 (time 1780338430.34 total 60.11 last 58.53)
 - open/close 4206 (time 1780338440.36 total 70.13 last 68.47)
 - open/close 4795 (time 1780338450.37 total 80.14 last 58.83)
 - open/close 5406 (time 1780338460.38 total 90.16 last 61.00)
 - open/close 5971 (time 1780338470.39 total 100.16 last 56.48)
 - open/close 6571 (time 1780338480.40 total 110.17 last 59.94)
 - open/close 7199 (time 1780338490.40 total 120.17 last 62.77)
 - open/close 7791 (time 1780338500.42 total 130.19 last 59.11)
 - open/close 8377 (time 1780338510.42 total 140.19 last 58.60)
 - open/close 8953 (time 1780338520.42 total 150.19 last 57.60)
 - open/close 9481 (time 1780338530.43 total 160.20 last 52.73)
total: 10000 open/close in 169.32 seconds: 59.06 ops/second
 - unlinked 0 (time 1780338544 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 568 (time 1780338666.20 total 10.01 last 56.75)
 - open/close 1122 (time 1780338676.21 total 20.02 last 55.33)
 - open/close 1715 (time 1780338686.24 total 30.05 last 59.15)
 - open/close 2307 (time 1780338696.24 total 40.05 last 59.16)
 - open/close 2907 (time 1780338706.25 total 50.06 last 59.93)
 - open/close 3523 (time 1780338716.27 total 60.08 last 61.53)
 - open/close 4150 (time 1780338726.27 total 70.08 last 62.66)
 - open/close 4763 (time 1780338736.28 total 80.09 last 61.26)
 - open/close 5375 (time 1780338746.31 total 90.12 last 61.02)
 - open/close 5998 (time 1780338756.33 total 100.14 last 62.18)
 - open/close 6571 (time 1780338766.34 total 110.15 last 57.26)
 - open/close 7147 (time 1780338776.35 total 120.16 last 57.50)
 - open/close 7719 (time 1780338786.37 total 130.18 last 57.11)
 - open/close 8265 (time 1780338796.38 total 140.19 last 54.53)
 - open/close 8852 (time 1780338806.40 total 150.21 last 58.60)
 - open/close 9493 (time 1780338816.40 total 160.21 last 64.09)
total: 10000 open/close in 168.73 seconds: 59.26 ops/second
 - unlinked 0 (time 1780338830 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 631 (time 1780338954.42 total 10.02 last 62.98)
 - open/close 1237 (time 1780338964.44 total 20.04 last 60.50)
 - open/close 1874 (time 1780338974.45 total 30.05 last 63.64)
 - open/close 2489 (time 1780338984.45 total 40.05 last 61.49)
 - open/close 3113 (time 1780338994.47 total 50.07 last 62.28)
 - open/close 3669 (time 1780339004.47 total 60.07 last 55.57)
 - open/close 4285 (time 1780339014.49 total 70.08 last 61.53)
 - open/close 4848 (time 1780339024.49 total 80.09 last 56.25)
 - open/close 5453 (time 1780339034.50 total 90.10 last 60.44)
 - open/close 6012 (time 1780339044.51 total 100.10 last 55.88)
 - open/close 6607 (time 1780339054.51 total 110.11 last 59.48)
 - open/close 7176 (time 1780339064.52 total 120.11 last 56.86)
 - open/close 7741 (time 1780339074.52 total 130.11 last 56.49)
 - open/close 8313 (time 1780339084.52 total 140.12 last 57.19)
 - open/close 8850 (time 1780339094.54 total 150.14 last 53.59)
 - open/close 9396 (time 1780339104.55 total 160.15 last 54.55)
 - open/close 9915 (time 1780339114.56 total 170.15 last 51.88)
total: 10000 open/close in 171.78 seconds: 58.21 ops/second
 - unlinked 0 (time 1780339121 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 483 (time 1780339240.93 total 10.02 last 48.19)
 - open/close 1011 (time 1780339250.96 total 20.05 last 52.66)
 - open/close 1586 (time 1780339260.97 total 30.06 last 57.42)
 - open/close 2203 (time 1780339270.98 total 40.07 last 61.63)
 - open/close 2803 (time 1780339281.00 total 50.09 last 59.90)
 - open/close 3400 (time 1780339291.00 total 60.10 last 59.67)
 - open/close 4010 (time 1780339301.01 total 70.10 last 60.96)
 - open/close 4628 (time 1780339311.02 total 80.11 last 61.75)
 - open/close 5264 (time 1780339321.02 total 90.11 last 63.58)
 - open/close 5852 (time 1780339331.03 total 100.12 last 58.78)
 - open/close 6477 (time 1780339341.03 total 110.12 last 62.47)
 - open/close 7044 (time 1780339351.04 total 120.14 last 56.63)
 - open/close 7687 (time 1780339361.05 total 130.14 last 64.29)
 - open/close 8260 (time 1780339371.05 total 140.14 last 57.27)
 - open/close 8826 (time 1780339381.05 total 150.15 last 56.59)
 - open/close 9436 (time 1780339391.06 total 160.15 last 60.97)
total: 10000 open/close in 168.95 seconds: 59.19 ops/second
 - unlinked 0 (time 1780339404 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 553 (time 1780339526.87 total 10.01 last 55.26)
 - open/close 944 (time 1780339536.88 total 20.01 last 39.07)
 - open/close 1357 (time 1780339546.92 total 30.06 last 41.12)
 - open/close 1948 (time 1780339556.93 total 40.06 last 59.07)
 - open/close 2535 (time 1780339566.94 total 50.07 last 58.67)
 - open/close 3096 (time 1780339576.94 total 60.07 last 56.09)
 - open/close 3698 (time 1780339586.94 total 70.07 last 60.20)
 - open/close 4352 (time 1780339596.94 total 80.07 last 65.40)
 - open/close 4995 (time 1780339606.96 total 90.09 last 64.16)
 - open/close 5609 (time 1780339616.97 total 100.10 last 61.34)
 - open/close 6158 (time 1780339626.98 total 110.12 last 54.82)
 - open/close 6742 (time 1780339636.99 total 120.12 last 58.37)
 - open/close 7331 (time 1780339647.00 total 130.13 last 58.84)
 - open/close 7934 (time 1780339657.00 total 140.14 last 60.28)
 - open/close 8566 (time 1780339667.01 total 150.14 last 63.18)
 - open/close 9145 (time 1780339677.01 total 160.15 last 57.86)
 - open/close 9730 (time 1780339687.01 total 170.15 last 58.50)
total: 10000 open/close in 174.74 seconds: 57.23 ops/second
 - unlinked 0 (time 1780339696 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 623 (time 1780339822.68 total 10.03 last 62.13)
 - open/close 1270 (time 1780339832.70 total 20.04 last 64.61)
 - open/close 1882 (time 1780339842.70 total 30.04 last 61.18)
 - open/close 2488 (time 1780339852.71 total 40.05 last 60.55)
 - open/close 3073 (time 1780339862.73 total 50.07 last 58.38)
 - open/close 3731 (time 1780339872.74 total 60.09 last 65.71)
 - open/close 4244 (time 1780339882.77 total 70.12 last 51.16)
 - open/close 4721 (time 1780339892.79 total 80.13 last 47.63)
 - open/close 5239 (time 1780339902.79 total 90.14 last 51.77)
 - open/close 5720 (time 1780339912.79 total 100.14 last 48.09)
 - open/close 6206 (time 1780339922.80 total 110.14 last 48.58)
 - open/close 6800 (time 1780339932.80 total 120.15 last 59.37)
 - open/close 7430 (time 1780339942.82 total 130.16 last 62.92)
 - open/close 8015 (time 1780339952.82 total 140.17 last 58.47)
 - open/close 8550 (time 1780339962.84 total 150.19 last 53.37)
 - open/close 9091 (time 1780339972.86 total 160.20 last 54.02)
 - open/close 9636 (time 1780339982.90 total 170.24 last 54.31)
total: 10000 open/close in 176.58 seconds: 56.63 ops/second
 - unlinked 0 (time 1780339993 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 622 (time 1780340117.68 total 10.01 last 62.15)
 - open/close 1218 (time 1780340127.70 total 20.02 last 59.51)
 - open/close 1845 (time 1780340137.72 total 30.05 last 62.56)
 - open/close 2469 (time 1780340147.73 total 40.05 last 62.35)
 - open/close 3053 (time 1780340157.75 total 50.07 last 58.29)
 - open/close 3660 (time 1780340167.76 total 60.08 last 60.64)
 - open/close 4237 (time 1780340177.76 total 70.09 last 57.67)
 - open/close 4863 (time 1780340187.78 total 80.10 last 62.49)
 - open/close 5462 (time 1780340197.78 total 90.11 last 59.88)
 - open/close 5955 (time 1780340207.79 total 100.11 last 49.27)
 - open/close 6380 (time 1780340217.80 total 110.12 last 42.46)
 - open/close 6969 (time 1780340227.81 total 120.13 last 58.83)
 - open/close 7610 (time 1780340237.83 total 130.15 last 63.98)
 - open/close 8227 (time 1780340247.83 total 140.16 last 61.68)
 - open/close 8845 (time 1780340257.83 total 150.16 last 61.80)
 - open/close 9439 (time 1780340267.84 total 160.17 last 59.34)
total: 10000 open/close in 169.51 seconds: 58.99 ops/second
 - unlinked 0 (time 1780340282 ; total 0 ; last 0)
total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second
 - open/close 608 (time 1780340413.41 total 10.02 last 60.70)
 - open/close 1147 (time 1780340423.43 total 20.03 last 53.81)
 - open/close 1776 (time 1780340433.44 total 30.04 last 62.87)
 - open/close 2364 (time 1780340443.45 total 40.05 last 58.74)
 - open/close 2950 (time 1780340453.45 total 50.06 last 58.55)
 - open/close 3508 (time 1780340463.46 total 60.06 last 55.78)
 - open/close 4024 (time 1780340473.46 total 70.06 last 51.58)
 - open/close 4639 (time 1780340483.47 total 80.07 last 61.48)
 - open/close 5213 (time 1780340493.47 total 90.08 last 57.36)
 - open/close 5716 (time 1780340503.50 total 100.10 last 50.20)
 - open/close 6260 (time 1780340513.51 total 110.11 last 54.34)
 - open/close 6790 (time 1780340523.51 total 120.11 last 52.99)
 - open/close 7297 (time 1780340533.52 total 130.12 last 50.63)
 - open/close 7920 (time 1780340543.54 total 140.14 last 62.22)
 - open/close 8489 (time 1780340553.55 total 150.15 last 56.85)
 - open/close 9086 (time 1780340563.55 total 160.15 last 59.68)
 - open/close 9649 (time 1780340573.56 total 170.17 last 56.22)
total: 10000 open/close in 175.41 seconds: 57.01 ops/second
 - unlinked 0 (time 1780340583 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 511 (time 1780340707.20 total 10.01 last 51.03)
 - open/close 1029 (time 1780340717.21 total 20.02 last 51.75)
 - open/close 1653 (time 1780340727.22 total 30.04 last 62.31)
 - open/close 2305 (time 1780340737.26 total 40.07 last 64.99)
 - open/close 2907 (time 1780340747.26 total 50.07 last 60.20)
 - open/close 3545 (time 1780340757.26 total 60.07 last 63.78)
 - open/close 4102 (time 1780340767.26 total 70.08 last 55.67)
 - open/close 4688 (time 1780340777.27 total 80.09 last 58.55)
 - open/close 5318 (time 1780340787.28 total 90.10 last 62.95)
 - open/close 5912 (time 1780340797.29 total 100.11 last 59.34)
 - open/close 6553 (time 1780340807.31 total 110.12 last 64.00)
 - open/close 7162 (time 1780340817.32 total 120.13 last 60.85)
 - open/close 7791 (time 1780340827.32 total 130.13 last 62.87)
 - open/close 8438 (time 1780340837.33 total 140.15 last 64.62)
 - open/close 9071 (time 1780340847.35 total 150.17 last 63.16)
 - open/close 9583 (time 1780340857.38 total 160.19 last 51.08)
total: 10000 open/close in 167.80 seconds: 59.59 ops/second
 - unlinked 0 (time 1780340870 ; total 0 ; last 0)
total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second
 - open/close 530 (time 1780340998.05 total 10.00 last 52.99)
 - open/close 1067 (time 1780341008.08 total 20.04 last 53.52)
 - open/close 1592 (time 1780341018.08 total 30.04 last 52.49)
 - open/close 2181 (time 1780341028.08 total 40.04 last 58.90)
 - open/close 2702 (time 1780341038.09 total 50.04 last 52.09)
 - open/close 3209 (time 1780341048.10 total 60.05 last 50.65)
 - open/close 3804 (time 1780341058.10 total 70.05 last 59.49)
 - open/close 4290 (time 1780341068.12 total 80.07 last 48.51)
 - open/close 4819 (time 1780341078.12 total 90.08 last 52.86)
 - open/close 5365 (time 1780341088.14 total 100.09 last 54.53)
 - open/close 5760 (time 1780341098.14 total 110.10 last 39.48)
 - open/close 6136 (time 1780341108.15 total 120.10 last 37.57)
 - open/close 6625 (time 1780341118.16 total 130.11 last 48.87)
 - open/close 7189 (time 1780341128.16 total 140.11 last 56.38)
 - open/close 7789 (time 1780341138.18 total 150.13 last 59.91)
 - open/close 8268 (time 1780341148.19 total 160.15 last 47.82)
 - open/close 8742 (time 1780341158.19 total 170.15 last 47.39)
 - open/close 9269 (time 1780341168.20 total 180.15 last 52.68)
 - open/close 9744 (time 1780341178.21 total 190.16 last 47.46)
total: 10000 open/close in 195.05 seconds: 51.27 ops/second
 - unlinked 0 (time 1780341190 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg451-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg451-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.151@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg451-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1
Started lustre-OST0000
oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg451-client.virtnet:  -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg451-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg451-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.151@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.151@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg451-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1
Started lustre-OST0000
oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg451-client.virtnet:  -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg451-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg451-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server
stop mds service on oleg451-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server

unloading modules via unload_modules_local on: 'oleg451-server'
oleg451-server: oleg451-server.virtnet: executing unload_modules_local
oleg451-server: 
modules unloaded.
pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2
pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2
pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2