== conf-sanity test 69: replace an OST with the same index ========================================================== 10:52:43 (1787583163)
start mds service on oleg624-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg624-server'
oleg624-server: oleg624-server.virtnet: executing load_modules_local
oleg624-server: Loading modules from /home/green/git/lustre-release/lustre
oleg624-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg624-server: detected 4 online CPUs by sysfs
oleg624-server: MODOPTS_LIBCFS=
oleg624-server: Force libcfs to create 2 CPU partitions
oleg624-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg624-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg624-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg624-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Started lustre-OST0000
oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg624-client.virtnet:  -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 678 (time 1787583251.77 total 10.00 last 67.79)
 - open/close 1383 (time 1787583261.78 total 20.01 last 70.44)
 - open/close 2087 (time 1787583271.78 total 30.01 last 70.39)
 - open/close 2808 (time 1787583281.80 total 40.02 last 72.00)
 - open/close 3553 (time 1787583291.80 total 50.03 last 74.47)
 - open/close 4272 (time 1787583301.81 total 60.04 last 71.82)
 - open/close 5001 (time 1787583311.81 total 70.04 last 72.89)
 - open/close 5730 (time 1787583321.81 total 80.04 last 72.90)
 - open/close 6387 (time 1787583331.82 total 90.05 last 65.65)
 - open/close 6966 (time 1787583341.82 total 100.05 last 57.89)
 - open/close 7597 (time 1787583351.83 total 110.06 last 63.03)
 - open/close 8217 (time 1787583361.84 total 120.07 last 61.99)
 - open/close 8817 (time 1787583371.84 total 130.07 last 59.98)
 - open/close 9527 (time 1787583381.85 total 140.08 last 70.94)
total: 10000 open/close in 146.54 seconds: 68.24 ops/second
 - unlinked 0 (time 1787583392 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 710 (time 1787583497.35 total 10.03 last 70.75)
 - open/close 1397 (time 1787583507.36 total 20.04 last 68.65)
 - open/close 2112 (time 1787583517.37 total 30.05 last 71.44)
 - open/close 2768 (time 1787583527.39 total 40.07 last 65.46)
 - open/close 3446 (time 1787583537.39 total 50.07 last 67.79)
 - open/close 4192 (time 1787583547.40 total 60.08 last 74.52)
 - open/close 4929 (time 1787583557.41 total 70.10 last 73.61)
 - open/close 5671 (time 1787583567.43 total 80.11 last 74.12)
 - open/close 6380 (time 1787583577.43 total 90.11 last 70.89)
 - open/close 7040 (time 1787583587.43 total 100.11 last 65.99)
 - open/close 7725 (time 1787583597.43 total 110.11 last 68.47)
 - open/close 8369 (time 1787583607.46 total 120.14 last 64.24)
 - open/close 9100 (time 1787583617.46 total 130.14 last 73.07)
 - open/close 9809 (time 1787583627.47 total 140.15 last 70.87)
total: 10000 open/close in 142.92 seconds: 69.97 ops/second
 - unlinked 0 (time 1787583634 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 542 (time 1787583735.06 total 10.00 last 54.18)
 - open/close 1326 (time 1787583745.06 total 20.00 last 78.39)
 - open/close 2044 (time 1787583755.08 total 30.03 last 71.64)
 - open/close 2678 (time 1787583765.09 total 40.03 last 63.35)
 - open/close 3372 (time 1787583775.10 total 50.04 last 69.35)
 - open/close 4049 (time 1787583785.11 total 60.05 last 67.62)
 - open/close 4763 (time 1787583795.12 total 70.06 last 71.32)
 - open/close 5439 (time 1787583805.13 total 80.07 last 67.53)
 - open/close 6072 (time 1787583815.13 total 90.07 last 63.30)
 - open/close 6807 (time 1787583825.14 total 100.08 last 73.44)
 - open/close 7477 (time 1787583835.15 total 110.09 last 66.95)
 - open/close 8188 (time 1787583845.15 total 120.10 last 71.05)
 - open/close 8858 (time 1787583855.17 total 130.12 last 66.86)
 - open/close 9492 (time 1787583865.20 total 140.14 last 63.26)
total: 10000 open/close in 148.36 seconds: 67.40 ops/second
 - unlinked 0 (time 1787583878 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 777 (time 1787583975.17 total 10.02 last 77.58)
 - open/close 1431 (time 1787583985.19 total 20.03 last 65.27)
 - open/close 2187 (time 1787583995.19 total 30.04 last 75.56)
 - open/close 2934 (time 1787584005.20 total 40.05 last 74.63)
 - open/close 3728 (time 1787584015.22 total 50.07 last 79.23)
 - open/close 4479 (time 1787584025.24 total 60.09 last 74.98)
 - open/close 5237 (time 1787584035.24 total 70.09 last 75.75)
 - open/close 6027 (time 1787584045.25 total 80.10 last 78.96)
 - open/close 6763 (time 1787584055.27 total 90.11 last 73.48)
 - open/close 7525 (time 1787584065.27 total 100.12 last 76.12)
 - open/close 8311 (time 1787584075.28 total 110.13 last 78.54)
 - open/close 9087 (time 1787584085.29 total 120.14 last 77.55)
 - open/close 9788 (time 1787584095.29 total 130.14 last 70.09)
total: 10000 open/close in 133.14 seconds: 75.11 ops/second
 - unlinked 0 (time 1787584102 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 812 (time 1787584198.64 total 10.01 last 81.14)
 - open/close 1541 (time 1787584208.65 total 20.01 last 72.86)
 - open/close 2209 (time 1787584218.65 total 30.01 last 66.79)
 - open/close 2951 (time 1787584228.66 total 40.02 last 74.14)
 - open/close 3768 (time 1787584238.66 total 50.03 last 81.66)
 - open/close 4634 (time 1787584248.68 total 60.04 last 86.46)
 - open/close 5357 (time 1787584258.69 total 70.05 last 72.23)
 - open/close 5994 (time 1787584268.70 total 80.06 last 63.66)
 - open/close 6596 (time 1787584278.71 total 90.07 last 60.13)
 - open/close 7287 (time 1787584288.72 total 100.09 last 68.98)
 - open/close 7837 (time 1787584298.73 total 110.09 last 54.97)
 - open/close 8258 (time 1787584308.74 total 120.10 last 42.08)
 - open/close 8673 (time 1787584318.74 total 130.10 last 41.49)
 - open/close 9138 (time 1787584328.75 total 140.11 last 46.46)
 - open/close 9693 (time 1787584338.76 total 150.12 last 55.45)
total: 10000 open/close in 155.13 seconds: 64.46 ops/second
 - unlinked 0 (time 1787584347 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 669 (time 1787584443.93 total 10.03 last 66.72)
 - open/close 1336 (time 1787584453.94 total 20.03 last 66.67)
 - open/close 2068 (time 1787584463.95 total 30.04 last 73.14)
 - open/close 2797 (time 1787584473.97 total 40.06 last 72.74)
 - open/close 3469 (time 1787584484.00 total 50.09 last 67.03)
 - open/close 4155 (time 1787584494.00 total 60.09 last 68.59)
 - open/close 4912 (time 1787584504.00 total 70.09 last 75.66)
 - open/close 5555 (time 1787584514.00 total 80.10 last 64.28)
 - open/close 6323 (time 1787584524.01 total 90.10 last 76.76)
 - open/close 7094 (time 1787584534.02 total 100.11 last 77.03)
 - open/close 7746 (time 1787584544.02 total 110.12 last 65.16)
 - open/close 8461 (time 1787584554.03 total 120.12 last 71.49)
 - open/close 9150 (time 1787584564.03 total 130.12 last 68.87)
 - open/close 9892 (time 1787584574.04 total 140.13 last 74.13)
total: 10000 open/close in 141.57 seconds: 70.64 ops/second
 - unlinked 0 (time 1787584579 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 776 (time 1787584681.57 total 10.01 last 77.54)
 - open/close 1512 (time 1787584691.57 total 20.01 last 73.57)
 - open/close 2238 (time 1787584701.57 total 30.01 last 72.59)
 - open/close 2964 (time 1787584711.58 total 40.02 last 72.52)
 - open/close 3730 (time 1787584721.60 total 50.04 last 76.47)
 - open/close 4443 (time 1787584731.63 total 60.07 last 71.07)
 - open/close 5094 (time 1787584741.64 total 70.08 last 65.07)
 - open/close 5722 (time 1787584751.64 total 80.08 last 62.78)
 - open/close 6382 (time 1787584761.64 total 90.09 last 65.97)
 - open/close 7066 (time 1787584771.66 total 100.10 last 68.30)
 - open/close 7789 (time 1787584781.66 total 110.10 last 72.29)
 - open/close 8511 (time 1787584791.66 total 120.11 last 72.18)
 - open/close 9271 (time 1787584801.67 total 130.11 last 75.94)
 - open/close 9886 (time 1787584811.67 total 140.12 last 61.49)
total: 10000 open/close in 142.00 seconds: 70.42 ops/second
 - unlinked 0 (time 1787584817 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 693 (time 1787584918.50 total 10.00 last 69.29)
 - open/close 1345 (time 1787584928.51 total 20.01 last 65.15)
 - open/close 2162 (time 1787584938.51 total 30.01 last 81.69)
 - open/close 2888 (time 1787584948.51 total 40.01 last 72.59)
 - open/close 3632 (time 1787584958.52 total 50.02 last 74.36)
 - open/close 4365 (time 1787584968.52 total 60.02 last 73.28)
 - open/close 5097 (time 1787584978.53 total 70.03 last 73.15)
 - open/close 5830 (time 1787584988.54 total 80.04 last 73.25)
 - open/close 6509 (time 1787584998.55 total 90.04 last 67.84)
 - open/close 7234 (time 1787585008.56 total 100.06 last 72.40)
 - open/close 7978 (time 1787585018.57 total 110.07 last 74.36)
 - open/close 8658 (time 1787585028.57 total 120.07 last 67.95)
 - open/close 9311 (time 1787585038.58 total 130.08 last 65.25)
 - open/close 9950 (time 1787585048.60 total 140.10 last 63.79)
total: 10000 open/close in 140.82 seconds: 71.01 ops/second
 - unlinked 0 (time 1787585053 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 610 (time 1787585157.01 total 10.00 last 60.98)
 - open/close 1289 (time 1787585167.02 total 20.01 last 67.84)
 - open/close 1986 (time 1787585177.05 total 30.04 last 69.52)
 - open/close 2699 (time 1787585187.06 total 40.05 last 71.21)
 - open/close 3415 (time 1787585197.08 total 50.07 last 71.50)
 - open/close 4110 (time 1787585207.08 total 60.07 last 69.45)
 - open/close 4825 (time 1787585217.10 total 70.08 last 71.42)
 - open/close 5557 (time 1787585227.13 total 80.12 last 72.95)
 - open/close 6253 (time 1787585237.13 total 90.12 last 69.58)
 - open/close 6922 (time 1787585247.13 total 100.12 last 66.89)
 - open/close 7654 (time 1787585257.15 total 110.14 last 73.10)
 - open/close 8370 (time 1787585267.16 total 120.15 last 71.51)
 - open/close 9075 (time 1787585277.17 total 130.16 last 70.45)
 - open/close 9797 (time 1787585287.19 total 140.17 last 72.07)
total: 10000 open/close in 142.81 seconds: 70.02 ops/second
 - unlinked 0 (time 1787585293 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 576 (time 1787585392.56 total 10.02 last 57.51)
 - open/close 1045 (time 1787585402.59 total 20.05 last 46.74)
 - open/close 1637 (time 1787585412.60 total 30.06 last 59.14)
 - open/close 2328 (time 1787585422.62 total 40.08 last 68.98)
 - open/close 3039 (time 1787585432.63 total 50.09 last 71.04)
 - open/close 3774 (time 1787585442.63 total 60.09 last 73.47)
 - open/close 4519 (time 1787585452.64 total 70.10 last 74.45)
 - open/close 5234 (time 1787585462.67 total 80.13 last 71.31)
 - open/close 5943 (time 1787585472.67 total 90.13 last 70.89)
 - open/close 6710 (time 1787585482.68 total 100.14 last 76.62)
 - open/close 7447 (time 1787585492.68 total 110.14 last 73.67)
 - open/close 8224 (time 1787585502.70 total 120.15 last 77.62)
 - open/close 8875 (time 1787585512.72 total 130.17 last 64.97)
 - open/close 9470 (time 1787585522.72 total 140.18 last 59.45)
total: 10000 open/close in 149.23 seconds: 67.01 ops/second
 - unlinked 0 (time 1787585538 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg624-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg624-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg624-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Started lustre-OST0000
oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg624-server: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg624-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg624-client.virtnet:  -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre
On OST0, 16390 used inodes rc=0
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg624-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg624-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.124@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.124@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.206.124@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg624-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Started lustre-OST0000
oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg624-server: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg624-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg624-client.virtnet:  -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg624-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg624-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server
stop mds service on oleg624-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server

unloading modules via unload_modules_local on: 'oleg624-server'
oleg624-server: oleg624-server.virtnet: executing unload_modules_local
oleg624-server: 
modules unloaded.
pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2
pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2