== conf-sanity test 69: replace an OST with the same index ========================================================== 19:32:48 (1789083168)
start mds service on oleg131-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg131-server'
oleg131-server: oleg131-server.virtnet: executing load_modules_local
oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg131-server: Loading modules from /home/green/git/lustre-release/lustre
oleg131-server: detected 4 online CPUs by sysfs
oleg131-server: MODOPTS_LIBCFS=
oleg131-server: Force libcfs to create 2 CPU partitions
oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 654 (time 1789083263.85 total 10.01 last 65.36)
 - open/close 1281 (time 1789083273.87 total 20.03 last 62.58)
 - open/close 2065 (time 1789083283.87 total 30.03 last 78.37)
 - open/close 2752 (time 1789083293.88 total 40.03 last 68.67)
 - open/close 3370 (time 1789083303.90 total 50.06 last 61.66)
 - open/close 3993 (time 1789083313.90 total 60.06 last 62.29)
 - open/close 4681 (time 1789083323.90 total 70.06 last 68.79)
 - open/close 5358 (time 1789083333.94 total 80.09 last 67.46)
 - open/close 5999 (time 1789083343.94 total 90.10 last 64.09)
 - open/close 6611 (time 1789083353.96 total 100.11 last 61.10)
 - open/close 7213 (time 1789083363.97 total 110.12 last 60.13)
 - open/close 7801 (time 1789083373.98 total 120.13 last 58.77)
 - open/close 8494 (time 1789083383.98 total 130.14 last 69.25)
 - open/close 9207 (time 1789083393.99 total 140.14 last 71.28)
 - open/close 9850 (time 1789083403.99 total 150.14 last 64.28)
total: 10000 open/close in 152.71 seconds: 65.49 ops/second
 - unlinked 0 (time 1789083411 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 690 (time 1789083532.51 total 10.00 last 68.99)
 - open/close 1479 (time 1789083542.51 total 20.01 last 78.85)
 - open/close 2202 (time 1789083552.52 total 30.02 last 72.22)
 - open/close 3009 (time 1789083562.52 total 40.02 last 80.68)
 - open/close 3698 (time 1789083572.53 total 50.03 last 68.83)
 - open/close 4356 (time 1789083582.54 total 60.04 last 65.77)
 - open/close 4892 (time 1789083592.56 total 70.06 last 53.49)
 - open/close 5447 (time 1789083602.58 total 80.07 last 55.41)
 - open/close 6047 (time 1789083612.58 total 90.08 last 59.98)
 - open/close 6843 (time 1789083622.60 total 100.10 last 79.40)
 - open/close 7598 (time 1789083632.62 total 110.12 last 75.38)
 - open/close 8396 (time 1789083642.63 total 120.12 last 79.74)
 - open/close 9100 (time 1789083652.64 total 130.13 last 70.34)
 - open/close 9895 (time 1789083662.65 total 140.14 last 79.41)
total: 10000 open/close in 141.89 seconds: 70.48 ops/second
 - unlinked 0 (time 1789083670 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 512 (time 1789083790.86 total 10.00 last 51.17)
 - open/close 1109 (time 1789083800.86 total 20.01 last 59.69)
 - open/close 1832 (time 1789083810.86 total 30.01 last 72.27)
 - open/close 2383 (time 1789083820.87 total 40.02 last 55.07)
 - open/close 3095 (time 1789083830.90 total 50.05 last 70.96)
 - open/close 3657 (time 1789083840.90 total 60.05 last 56.19)
 - open/close 4273 (time 1789083850.92 total 70.07 last 61.47)
 - open/close 4990 (time 1789083860.94 total 80.08 last 71.63)
 - open/close 5679 (time 1789083870.94 total 90.09 last 68.85)
 - open/close 6404 (time 1789083880.94 total 100.09 last 72.49)
 - open/close 7093 (time 1789083890.94 total 110.09 last 68.90)
 - open/close 7804 (time 1789083900.96 total 120.11 last 70.99)
 - open/close 8492 (time 1789083910.97 total 130.12 last 68.74)
 - open/close 9105 (time 1789083920.98 total 140.13 last 61.23)
 - open/close 9684 (time 1789083930.98 total 150.13 last 57.90)
total: 10000 open/close in 155.50 seconds: 64.31 ops/second
 - unlinked 0 (time 1789083941 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 658 (time 1789084053.26 total 10.01 last 65.76)
 - open/close 1298 (time 1789084063.27 total 20.02 last 63.93)
 - open/close 1967 (time 1789084073.27 total 30.02 last 66.89)
 - open/close 2580 (time 1789084083.28 total 40.02 last 61.28)
 - open/close 3209 (time 1789084093.30 total 50.04 last 62.76)
 - open/close 3770 (time 1789084103.30 total 60.05 last 56.06)
 - open/close 4433 (time 1789084113.31 total 70.06 last 66.25)
 - open/close 5144 (time 1789084123.32 total 80.06 last 71.06)
 - open/close 5860 (time 1789084133.33 total 90.07 last 71.52)
 - open/close 6505 (time 1789084143.35 total 100.09 last 64.38)
 - open/close 7247 (time 1789084153.36 total 110.10 last 74.12)
 - open/close 7893 (time 1789084163.38 total 120.13 last 64.42)
 - open/close 8552 (time 1789084173.39 total 130.14 last 65.87)
 - open/close 9216 (time 1789084183.40 total 140.15 last 66.31)
 - open/close 9829 (time 1789084193.40 total 150.15 last 61.29)
total: 10000 open/close in 152.68 seconds: 65.49 ops/second
 - unlinked 0 (time 1789084201 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 600 (time 1789084307.82 total 10.04 last 59.76)
 - open/close 1249 (time 1789084317.83 total 20.05 last 64.84)
 - open/close 1948 (time 1789084327.84 total 30.05 last 69.87)
 - open/close 2662 (time 1789084337.84 total 40.05 last 71.39)
 - open/close 3336 (time 1789084347.85 total 50.06 last 67.34)
 - open/close 3934 (time 1789084357.86 total 60.07 last 59.75)
 - open/close 4519 (time 1789084367.87 total 70.09 last 58.41)
 - open/close 5150 (time 1789084377.88 total 80.09 last 63.06)
 - open/close 5815 (time 1789084387.88 total 90.10 last 66.47)
 - open/close 6543 (time 1789084397.89 total 100.11 last 72.74)
 - open/close 7262 (time 1789084407.90 total 110.12 last 71.83)
 - open/close 7924 (time 1789084417.91 total 120.12 last 66.15)
 - open/close 8599 (time 1789084427.92 total 130.14 last 67.39)
 - open/close 9145 (time 1789084437.93 total 140.14 last 54.59)
 - open/close 9868 (time 1789084447.94 total 150.15 last 72.22)
total: 10000 open/close in 152.17 seconds: 65.72 ops/second
 - unlinked 0 (time 1789084454 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 607 (time 1789084564.82 total 10.00 last 60.68)
 - open/close 1397 (time 1789084574.82 total 20.01 last 78.97)
 - open/close 2083 (time 1789084584.83 total 30.02 last 68.54)
 - open/close 2719 (time 1789084594.84 total 40.02 last 63.57)
 - open/close 3270 (time 1789084604.84 total 50.02 last 55.08)
 - open/close 3829 (time 1789084614.85 total 60.03 last 55.86)
 - open/close 4513 (time 1789084624.85 total 70.04 last 68.36)
 - open/close 5194 (time 1789084634.87 total 80.05 last 67.99)
 - open/close 5844 (time 1789084644.88 total 90.07 last 64.90)
 - open/close 6492 (time 1789084654.90 total 100.08 last 64.71)
 - open/close 7209 (time 1789084664.90 total 110.08 last 71.69)
 - open/close 7922 (time 1789084674.92 total 120.10 last 71.15)
 - open/close 8553 (time 1789084684.93 total 130.11 last 63.05)
 - open/close 9241 (time 1789084694.93 total 140.11 last 68.78)
 - open/close 9914 (time 1789084704.94 total 150.12 last 67.23)
total: 10000 open/close in 151.82 seconds: 65.87 ops/second
 - unlinked 0 (time 1789084711 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 725 (time 1789084817.70 total 10.01 last 72.45)
 - open/close 1424 (time 1789084827.73 total 20.04 last 69.69)
 - open/close 2130 (time 1789084837.74 total 30.04 last 70.55)
 - open/close 2869 (time 1789084847.74 total 40.05 last 73.89)
 - open/close 3645 (time 1789084857.74 total 50.05 last 77.60)
 - open/close 4339 (time 1789084867.75 total 60.05 last 69.34)
 - open/close 5052 (time 1789084877.76 total 70.07 last 71.21)
 - open/close 5757 (time 1789084887.77 total 80.08 last 70.41)
 - open/close 6334 (time 1789084897.79 total 90.09 last 57.61)
 - open/close 6953 (time 1789084907.81 total 100.12 last 61.78)
 - open/close 7626 (time 1789084917.82 total 110.13 last 67.22)
 - open/close 8176 (time 1789084927.83 total 120.14 last 54.95)
 - open/close 8705 (time 1789084937.83 total 130.14 last 52.88)
 - open/close 9331 (time 1789084947.84 total 140.14 last 62.57)
 - open/close 9904 (time 1789084957.85 total 150.15 last 57.25)
total: 10000 open/close in 151.60 seconds: 65.96 ops/second
 - unlinked 0 (time 1789084964 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 682 (time 1789085081.89 total 10.00 last 68.18)
 - open/close 1426 (time 1789085091.93 total 20.04 last 74.12)
 - open/close 2069 (time 1789085101.93 total 30.05 last 64.27)
 - open/close 2800 (time 1789085111.94 total 40.06 last 73.01)
 - open/close 3396 (time 1789085121.96 total 50.07 last 59.53)
 - open/close 4027 (time 1789085131.97 total 60.08 last 63.04)
 - open/close 4665 (time 1789085141.97 total 70.09 last 63.74)
 - open/close 5303 (time 1789085151.98 total 80.10 last 63.76)
 - open/close 6026 (time 1789085161.99 total 90.10 last 72.24)
 - open/close 6679 (time 1789085172.00 total 100.11 last 65.25)
 - open/close 7315 (time 1789085182.00 total 110.12 last 63.56)
 - open/close 7939 (time 1789085192.02 total 120.13 last 62.31)
 - open/close 8594 (time 1789085202.02 total 130.14 last 65.46)
 - open/close 9276 (time 1789085212.03 total 140.15 last 68.14)
 - open/close 9912 (time 1789085222.03 total 150.15 last 63.60)
total: 10000 open/close in 151.93 seconds: 65.82 ops/second
 - unlinked 0 (time 1789085229 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 589 (time 1789085341.84 total 10.01 last 58.87)
 - open/close 1258 (time 1789085351.85 total 20.02 last 66.82)
 - open/close 1909 (time 1789085361.86 total 30.03 last 65.00)
 - open/close 2593 (time 1789085371.87 total 40.04 last 68.36)
 - open/close 3262 (time 1789085381.88 total 50.05 last 66.84)
 - open/close 3853 (time 1789085391.88 total 60.05 last 59.10)
 - open/close 4415 (time 1789085401.89 total 70.05 last 56.16)
 - open/close 5004 (time 1789085411.91 total 80.08 last 58.75)
 - open/close 5526 (time 1789085421.94 total 90.10 last 52.08)
 - open/close 6243 (time 1789085431.95 total 100.12 last 71.60)
 - open/close 6840 (time 1789085441.95 total 110.12 last 59.69)
 - open/close 7470 (time 1789085451.97 total 120.13 last 62.90)
 - open/close 8181 (time 1789085461.99 total 130.16 last 70.93)
 - open/close 8797 (time 1789085471.99 total 140.16 last 61.59)
 - open/close 9518 (time 1789085482.00 total 150.17 last 72.03)
total: 10000 open/close in 158.90 seconds: 62.93 ops/second
 - unlinked 0 (time 1789085495 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 646 (time 1789085614.76 total 10.00 last 64.57)
 - open/close 1291 (time 1789085624.76 total 20.01 last 64.46)
 - open/close 1971 (time 1789085634.77 total 30.01 last 67.99)
 - open/close 2671 (time 1789085644.80 total 40.04 last 69.79)
 - open/close 3354 (time 1789085654.80 total 50.05 last 68.24)
 - open/close 3989 (time 1789085664.81 total 60.05 last 63.48)
 - open/close 4633 (time 1789085674.81 total 70.06 last 64.39)
 - open/close 5141 (time 1789085684.96 total 80.20 last 50.07)
 - open/close 5618 (time 1789085694.99 total 90.24 last 47.53)
 - open/close 6260 (time 1789085705.00 total 100.24 last 64.16)
 - open/close 6895 (time 1789085715.00 total 110.25 last 63.48)
 - open/close 7651 (time 1789085725.00 total 120.25 last 75.57)
 - open/close 8301 (time 1789085735.01 total 130.26 last 64.94)
 - open/close 8988 (time 1789085745.02 total 140.27 last 68.66)
 - open/close 9643 (time 1789085755.02 total 150.27 last 65.47)
total: 10000 open/close in 154.76 seconds: 64.62 ops/second
 - unlinked 0 (time 1789085763 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.131@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server
stop mds service on oleg131-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server

unloading modules via unload_modules_local on: 'oleg131-server'
oleg131-server: oleg131-server.virtnet: executing unload_modules_local
oleg131-server: 
modules unloaded.
pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2