== conf-sanity test 69: replace an OST with the same index ========================================================== 08:24:26 (1789129466)
start mds service on oleg459-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing load_modules_local
oleg459-server: Loading modules from /home/green/git/lustre-release/lustre
oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg459-server: detected 4 online CPUs by sysfs
oleg459-server: MODOPTS_LIBCFS=
oleg459-server: Force libcfs to create 2 CPU partitions
oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg459-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 633 (time 1789129562.52 total 10.02 last 63.18)
 - open/close 1253 (time 1789129572.52 total 20.02 last 61.99)
 - open/close 1841 (time 1789129582.53 total 30.03 last 58.77)
 - open/close 2508 (time 1789129592.54 total 40.04 last 66.62)
 - open/close 3087 (time 1789129602.55 total 50.04 last 57.88)
 - open/close 3736 (time 1789129612.55 total 60.04 last 64.89)
 - open/close 4262 (time 1789129622.55 total 70.05 last 52.58)
 - open/close 4865 (time 1789129632.56 total 80.06 last 60.25)
 - open/close 5503 (time 1789129642.56 total 90.06 last 63.77)
 - open/close 6155 (time 1789129652.58 total 100.08 last 65.09)
 - open/close 6755 (time 1789129662.59 total 110.08 last 59.96)
 - open/close 7432 (time 1789129672.59 total 120.09 last 67.67)
 - open/close 8093 (time 1789129682.61 total 130.11 last 65.98)
 - open/close 8717 (time 1789129692.64 total 140.13 last 62.23)
 - open/close 9220 (time 1789129702.65 total 150.14 last 50.26)
 - open/close 9695 (time 1789129712.66 total 160.16 last 47.44)
total: 10000 open/close in 165.11 seconds: 60.57 ops/second
 - unlinked 0 (time 1789129722 ; total 0 ; last 0)
total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second
 - open/close 591 (time 1789129855.12 total 10.01 last 59.02)
 - open/close 1160 (time 1789129865.15 total 20.04 last 56.74)
 - open/close 1737 (time 1789129875.16 total 30.05 last 57.67)
 - open/close 2321 (time 1789129885.16 total 40.05 last 58.38)
 - open/close 2902 (time 1789129895.16 total 50.05 last 58.08)
 - open/close 3368 (time 1789129905.17 total 60.06 last 46.56)
 - open/close 3967 (time 1789129915.18 total 70.07 last 59.86)
 - open/close 4596 (time 1789129925.18 total 80.07 last 62.86)
 - open/close 5196 (time 1789129935.18 total 90.08 last 60.00)
 - open/close 5729 (time 1789129945.20 total 100.09 last 53.22)
 - open/close 6259 (time 1789129955.23 total 110.12 last 52.82)
 - open/close 6912 (time 1789129965.25 total 120.14 last 65.19)
 - open/close 7538 (time 1789129975.26 total 130.15 last 62.53)
 - open/close 8129 (time 1789129985.27 total 140.16 last 59.04)
 - open/close 8720 (time 1789129995.27 total 150.16 last 59.07)
 - open/close 9314 (time 1789130005.28 total 160.17 last 59.37)
 - open/close 9897 (time 1789130015.29 total 170.18 last 58.23)
total: 10000 open/close in 172.42 seconds: 58.00 ops/second
 - unlinked 0 (time 1789130022 ; total 0 ; last 0)
total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second
 - open/close 553 (time 1789130154.11 total 10.03 last 55.12)
 - open/close 1059 (time 1789130164.11 total 20.04 last 50.59)
 - open/close 1530 (time 1789130174.11 total 30.04 last 47.09)
 - open/close 2080 (time 1789130184.11 total 40.04 last 55.00)
 - open/close 2598 (time 1789130194.12 total 50.05 last 51.73)
 - open/close 3132 (time 1789130204.15 total 60.08 last 53.26)
 - open/close 3694 (time 1789130214.17 total 70.09 last 56.12)
 - open/close 4252 (time 1789130224.18 total 80.11 last 55.70)
 - open/close 4815 (time 1789130234.22 total 90.14 last 56.10)
 - open/close 5371 (time 1789130244.23 total 100.15 last 55.55)
 - open/close 5872 (time 1789130254.23 total 110.16 last 50.06)
 - open/close 6375 (time 1789130264.25 total 120.17 last 50.24)
 - open/close 6984 (time 1789130274.25 total 130.18 last 60.87)
 - open/close 7553 (time 1789130284.25 total 140.18 last 56.89)
 - open/close 8158 (time 1789130294.25 total 150.18 last 60.49)
 - open/close 8728 (time 1789130304.26 total 160.18 last 56.98)
 - open/close 9310 (time 1789130314.27 total 170.19 last 58.15)
 - open/close 9902 (time 1789130324.27 total 180.20 last 59.16)
total: 10000 open/close in 181.94 seconds: 54.96 ops/second
 - unlinked 0 (time 1789130330 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 580 (time 1789130452.00 total 10.00 last 57.97)
 - open/close 1163 (time 1789130462.01 total 20.01 last 58.25)
 - open/close 1734 (time 1789130472.01 total 30.02 last 57.06)
 - open/close 2200 (time 1789130482.03 total 40.03 last 46.55)
 - open/close 2758 (time 1789130492.07 total 50.07 last 55.57)
 - open/close 3299 (time 1789130502.11 total 60.11 last 53.89)
 - open/close 3849 (time 1789130512.14 total 70.14 last 54.83)
 - open/close 4415 (time 1789130522.16 total 80.17 last 56.46)
 - open/close 4996 (time 1789130532.18 total 90.18 last 58.02)
 - open/close 5583 (time 1789130542.19 total 100.19 last 58.63)
 - open/close 6186 (time 1789130552.23 total 110.24 last 60.03)
 - open/close 6729 (time 1789130562.24 total 120.24 last 54.27)
 - open/close 7300 (time 1789130572.24 total 130.24 last 57.09)
 - open/close 7838 (time 1789130582.24 total 140.24 last 53.80)
 - open/close 8391 (time 1789130592.25 total 150.25 last 55.24)
 - open/close 8978 (time 1789130602.26 total 160.26 last 58.67)
 - open/close 9507 (time 1789130612.26 total 170.27 last 52.87)
total: 10000 open/close in 179.72 seconds: 55.64 ops/second
 - unlinked 0 (time 1789130627 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 574 (time 1789130742.28 total 10.01 last 57.35)
 - open/close 1154 (time 1789130752.28 total 20.01 last 58.00)
 - open/close 1746 (time 1789130762.30 total 30.02 last 59.13)
 - open/close 2319 (time 1789130772.31 total 40.03 last 57.23)
 - open/close 2906 (time 1789130782.31 total 50.04 last 58.69)
 - open/close 3447 (time 1789130792.32 total 60.04 last 54.06)
 - open/close 4013 (time 1789130802.33 total 70.05 last 56.55)
 - open/close 4570 (time 1789130812.33 total 80.06 last 55.66)
 - open/close 5174 (time 1789130822.36 total 90.09 last 60.22)
 - open/close 5837 (time 1789130832.36 total 100.09 last 66.29)
 - open/close 6496 (time 1789130842.37 total 110.09 last 65.87)
 - open/close 7103 (time 1789130852.38 total 120.11 last 60.61)
 - open/close 7788 (time 1789130862.39 total 130.12 last 68.42)
 - open/close 8401 (time 1789130872.40 total 140.12 last 61.28)
 - open/close 9060 (time 1789130882.40 total 150.13 last 65.85)
 - open/close 9694 (time 1789130892.41 total 160.13 last 63.39)
total: 10000 open/close in 164.65 seconds: 60.73 ops/second
 - unlinked 0 (time 1789130901 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 697 (time 1789131016.30 total 10.01 last 69.65)
 - open/close 1325 (time 1789131026.34 total 20.05 last 62.51)
 - open/close 1955 (time 1789131036.36 total 30.07 last 62.91)
 - open/close 2565 (time 1789131046.37 total 40.08 last 60.94)
 - open/close 3267 (time 1789131056.37 total 50.08 last 70.19)
 - open/close 3990 (time 1789131066.38 total 60.09 last 72.21)
 - open/close 4665 (time 1789131076.39 total 70.10 last 67.46)
 - open/close 5375 (time 1789131086.40 total 80.11 last 70.92)
 - open/close 6081 (time 1789131096.41 total 90.12 last 70.50)
 - open/close 6744 (time 1789131106.41 total 100.13 last 66.29)
 - open/close 7314 (time 1789131116.43 total 110.14 last 56.92)
 - open/close 7866 (time 1789131126.43 total 120.15 last 55.17)
 - open/close 8452 (time 1789131136.44 total 130.15 last 58.56)
 - open/close 9054 (time 1789131146.45 total 140.16 last 60.15)
 - open/close 9726 (time 1789131156.45 total 150.17 last 67.16)
total: 10000 open/close in 153.87 seconds: 64.99 ops/second
 - unlinked 0 (time 1789131164 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 623 (time 1789131275.37 total 10.03 last 62.13)
 - open/close 1295 (time 1789131285.37 total 20.03 last 67.20)
 - open/close 1913 (time 1789131295.38 total 30.04 last 61.72)
 - open/close 2544 (time 1789131305.40 total 40.06 last 62.99)
 - open/close 3213 (time 1789131315.41 total 50.07 last 66.83)
 - open/close 3913 (time 1789131325.41 total 60.07 last 69.98)
 - open/close 4547 (time 1789131335.42 total 70.08 last 63.38)
 - open/close 5144 (time 1789131345.42 total 80.08 last 59.66)
 - open/close 5790 (time 1789131355.43 total 90.09 last 64.54)
 - open/close 6425 (time 1789131365.44 total 100.10 last 63.46)
 - open/close 7089 (time 1789131375.46 total 110.12 last 66.28)
 - open/close 7594 (time 1789131385.47 total 120.13 last 50.44)
 - open/close 8203 (time 1789131395.48 total 130.14 last 60.84)
 - open/close 8876 (time 1789131405.52 total 140.18 last 67.02)
 - open/close 9561 (time 1789131415.53 total 150.18 last 68.48)
total: 10000 open/close in 158.68 seconds: 63.02 ops/second
 - unlinked 0 (time 1789131429 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 603 (time 1789131542.91 total 10.01 last 60.26)
 - open/close 1258 (time 1789131552.91 total 20.01 last 65.46)
 - open/close 1885 (time 1789131562.92 total 30.01 last 62.69)
 - open/close 2614 (time 1789131572.92 total 40.02 last 72.86)
 - open/close 3298 (time 1789131582.94 total 50.03 last 68.30)
 - open/close 3941 (time 1789131592.94 total 60.04 last 64.25)
 - open/close 4561 (time 1789131602.95 total 70.04 last 61.99)
 - open/close 5201 (time 1789131612.95 total 80.05 last 63.95)
 - open/close 5893 (time 1789131622.96 total 90.05 last 69.18)
 - open/close 6610 (time 1789131632.96 total 100.06 last 71.67)
 - open/close 7277 (time 1789131642.97 total 110.07 last 66.64)
 - open/close 7962 (time 1789131652.99 total 120.09 last 68.37)
 - open/close 8576 (time 1789131663.00 total 130.10 last 61.33)
 - open/close 9175 (time 1789131673.01 total 140.11 last 59.84)
 - open/close 9884 (time 1789131683.02 total 150.12 last 70.83)
total: 10000 open/close in 152.08 seconds: 65.76 ops/second
 - unlinked 0 (time 1789131689 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 631 (time 1789131802.27 total 10.01 last 63.03)
 - open/close 1154 (time 1789131812.30 total 20.04 last 52.17)
 - open/close 1666 (time 1789131822.32 total 30.06 last 51.07)
 - open/close 2199 (time 1789131832.34 total 40.07 last 53.23)
 - open/close 2760 (time 1789131842.36 total 50.10 last 55.96)
 - open/close 3252 (time 1789131852.39 total 60.12 last 49.08)
 - open/close 3767 (time 1789131862.39 total 70.13 last 51.49)
 - open/close 4262 (time 1789131872.40 total 80.13 last 49.47)
 - open/close 4814 (time 1789131882.42 total 90.16 last 55.07)
 - open/close 5423 (time 1789131892.44 total 100.18 last 60.76)
 - open/close 6058 (time 1789131902.45 total 110.19 last 63.44)
 - open/close 6723 (time 1789131912.46 total 120.20 last 66.45)
 - open/close 7321 (time 1789131922.47 total 130.20 last 59.76)
 - open/close 7953 (time 1789131932.47 total 140.21 last 63.18)
 - open/close 8557 (time 1789131942.47 total 150.21 last 60.38)
 - open/close 9186 (time 1789131952.47 total 160.21 last 62.89)
 - open/close 9794 (time 1789131962.47 total 170.21 last 60.79)
total: 10000 open/close in 173.58 seconds: 57.61 ops/second
 - unlinked 0 (time 1789131970 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 583 (time 1789132083.40 total 10.02 last 58.16)
 - open/close 1275 (time 1789132093.40 total 20.02 last 69.20)
 - open/close 1913 (time 1789132103.40 total 30.03 last 63.79)
 - open/close 2606 (time 1789132113.41 total 40.03 last 69.27)
 - open/close 3295 (time 1789132123.41 total 50.03 last 68.86)
 - open/close 3939 (time 1789132133.42 total 60.04 last 64.37)
 - open/close 4625 (time 1789132143.43 total 70.05 last 68.54)
 - open/close 5226 (time 1789132153.43 total 80.05 last 60.10)
 - open/close 5858 (time 1789132163.44 total 90.06 last 63.13)
 - open/close 6577 (time 1789132173.44 total 100.06 last 71.87)
 - open/close 7231 (time 1789132183.45 total 110.07 last 65.32)
 - open/close 7888 (time 1789132193.47 total 120.09 last 65.62)
 - open/close 8459 (time 1789132203.47 total 130.09 last 57.06)
 - open/close 9192 (time 1789132213.49 total 140.11 last 73.16)
 - open/close 9860 (time 1789132223.50 total 150.12 last 66.76)
total: 10000 open/close in 152.30 seconds: 65.66 ops/second
 - unlinked 0 (time 1789132230 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
On OST0, 20327 used inodes rc=0
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server
stop mds service on oleg459-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server

unloading modules via unload_modules_local on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing unload_modules_local
oleg459-server: 
modules unloaded.
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2