== conf-sanity test 69: replace an OST with the same index ========================================================== 17:39:39 (1788903579)
start mds service on oleg256-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg256-server'
oleg256-server: oleg256-server.virtnet: executing load_modules_local
oleg256-server: Loading modules from /home/green/git/lustre-release/lustre
oleg256-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg256-server: detected 4 online CPUs by sysfs
oleg256-server: MODOPTS_LIBCFS=
oleg256-server: Force libcfs to create 2 CPU partitions
oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg256-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg256-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1
Started lustre-OST0000
oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg256-client.virtnet:  -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1207 (time 1788903646.28 total 10.00 last 120.69)
 - open/close 2299 (time 1788903656.29 total 20.01 last 109.08)
 - open/close 3462 (time 1788903666.29 total 30.01 last 116.28)
 - open/close 4599 (time 1788903676.29 total 40.02 last 113.64)
 - open/close 5740 (time 1788903686.30 total 50.02 last 114.07)
 - open/close 6909 (time 1788903696.30 total 60.02 last 116.89)
 - open/close 8034 (time 1788903706.30 total 70.03 last 112.44)
 - open/close 9110 (time 1788903716.31 total 80.03 last 107.57)
total: 10000 open/close in 88.32 seconds: 113.22 ops/second
 - unlinked 0 (time 1788903727 ; total 0 ; last 0)
total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second
 - open/close 1283 (time 1788903786.99 total 10.00 last 128.24)
 - open/close 2494 (time 1788903797.00 total 20.02 last 120.97)
 - open/close 3840 (time 1788903807.01 total 30.03 last 134.46)
 - open/close 4927 (time 1788903817.01 total 40.03 last 108.63)
 - open/close 6277 (time 1788903827.02 total 50.04 last 134.93)
 - open/close 8337 (time 1788903837.02 total 60.04 last 205.97)
total: 10000 open/close in 68.74 seconds: 145.48 ops/second
 - unlinked 0 (time 1788903847 ; total 0 ; last 0)
total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second
 - open/close 1974 (time 1788903893.34 total 10.00 last 197.38)
 - open/close 3560 (time 1788903903.34 total 20.01 last 158.54)
 - open/close 5569 (time 1788903913.34 total 30.01 last 200.83)
 - open/close 7260 (time 1788903923.35 total 40.01 last 169.00)
 - open/close 8744 (time 1788903933.35 total 50.01 last 148.39)
total: 10000 open/close in 57.40 seconds: 174.23 ops/second
 - unlinked 0 (time 1788903942 ; total 0 ; last 0)
total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second
 - open/close 1578 (time 1788903986.64 total 10.01 last 157.69)
 - open/close 3323 (time 1788903996.64 total 20.01 last 174.44)
 - open/close 5392 (time 1788904006.65 total 30.02 last 206.78)
 - open/close 7129 (time 1788904016.65 total 40.02 last 173.67)
 - open/close 8910 (time 1788904026.66 total 50.03 last 177.96)
total: 10000 open/close in 55.17 seconds: 181.24 ops/second
 - unlinked 0 (time 1788904033 ; total 0 ; last 0)
total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second
 - open/close 1967 (time 1788904080.03 total 10.00 last 196.62)
 - open/close 3863 (time 1788904090.04 total 20.01 last 189.54)
 - open/close 5740 (time 1788904100.04 total 30.02 last 187.53)
 - open/close 7568 (time 1788904110.05 total 40.02 last 182.67)
 - open/close 9309 (time 1788904120.05 total 50.03 last 174.06)
total: 10000 open/close in 54.34 seconds: 184.03 ops/second
 - unlinked 0 (time 1788904126 ; total 0 ; last 0)
total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second
 - open/close 965 (time 1788904174.16 total 10.01 last 96.40)
 - open/close 1832 (time 1788904184.17 total 20.02 last 86.64)
 - open/close 2715 (time 1788904194.17 total 30.02 last 88.26)
 - open/close 3930 (time 1788904204.17 total 40.02 last 121.50)
 - open/close 5201 (time 1788904214.18 total 50.03 last 126.97)
 - open/close 6163 (time 1788904224.19 total 60.04 last 96.14)
 - open/close 6731 (time 1788904234.20 total 70.05 last 56.75)
 - open/close 7352 (time 1788904244.20 total 80.05 last 62.09)
 - open/close 8040 (time 1788904254.22 total 90.07 last 68.65)
 - open/close 8444 (time 1788904264.23 total 100.08 last 40.38)
 - open/close 8710 (time 1788904274.53 total 110.38 last 25.82)
 - open/close 8909 (time 1788904284.55 total 120.40 last 19.86)
 - open/close 9155 (time 1788904294.55 total 130.40 last 24.60)
 - open/close 9540 (time 1788904304.61 total 140.46 last 38.26)
 - open/close 9841 (time 1788904314.62 total 150.47 last 30.08)
total: 10000 open/close in 153.65 seconds: 65.08 ops/second
 - unlinked 0 (time 1788904322 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 732 (time 1788904429.77 total 10.01 last 73.13)
 - open/close 1420 (time 1788904439.78 total 20.02 last 68.74)
 - open/close 2094 (time 1788904449.78 total 30.02 last 67.40)
 - open/close 2804 (time 1788904459.79 total 40.02 last 70.96)
 - open/close 3568 (time 1788904469.80 total 50.03 last 76.34)
 - open/close 4371 (time 1788904479.80 total 60.03 last 80.28)
 - open/close 5075 (time 1788904489.80 total 70.04 last 70.37)
 - open/close 5784 (time 1788904499.80 total 80.04 last 70.89)
 - open/close 6466 (time 1788904509.81 total 90.04 last 68.20)
 - open/close 7170 (time 1788904519.82 total 100.05 last 70.32)
 - open/close 7785 (time 1788904529.83 total 110.06 last 61.44)
 - open/close 8494 (time 1788904539.83 total 120.07 last 70.88)
 - open/close 9156 (time 1788904549.84 total 130.08 last 66.12)
 - open/close 9735 (time 1788904559.87 total 140.11 last 57.73)
total: 10000 open/close in 144.49 seconds: 69.21 ops/second
 - unlinked 0 (time 1788904569 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 499 (time 1788904680.79 total 10.01 last 49.87)
 - open/close 1043 (time 1788904690.79 total 20.01 last 54.38)
 - open/close 1644 (time 1788904700.82 total 30.03 last 59.97)
 - open/close 2147 (time 1788904710.83 total 40.04 last 50.24)
 - open/close 2653 (time 1788904720.84 total 50.06 last 50.55)
 - open/close 3274 (time 1788904730.84 total 60.06 last 62.10)
 - open/close 4109 (time 1788904740.84 total 70.06 last 83.47)
 - open/close 4869 (time 1788904750.85 total 80.07 last 75.92)
 - open/close 5686 (time 1788904760.86 total 90.07 last 81.67)
 - open/close 6527 (time 1788904770.86 total 100.08 last 84.05)
 - open/close 7431 (time 1788904780.87 total 110.08 last 90.36)
 - open/close 8384 (time 1788904790.87 total 120.09 last 95.22)
 - open/close 9235 (time 1788904800.88 total 130.10 last 85.06)
total: 10000 open/close in 139.18 seconds: 71.85 ops/second
 - unlinked 0 (time 1788904813 ; total 0 ; last 0)
total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second
 - open/close 1037 (time 1788904889.24 total 10.00 last 103.65)
 - open/close 2116 (time 1788904899.24 total 20.01 last 107.87)
 - open/close 3327 (time 1788904909.25 total 30.01 last 121.04)
 - open/close 4465 (time 1788904919.25 total 40.01 last 113.80)
 - open/close 5568 (time 1788904929.26 total 50.02 last 110.21)
 - open/close 6744 (time 1788904939.27 total 60.03 last 117.46)
 - open/close 7944 (time 1788904949.28 total 70.04 last 119.89)
 - open/close 9177 (time 1788904959.28 total 80.05 last 123.25)
total: 10000 open/close in 87.60 seconds: 114.16 ops/second
 - unlinked 0 (time 1788904969 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1135 (time 1788905031.38 total 10.01 last 113.42)
 - open/close 2440 (time 1788905041.38 total 20.01 last 130.45)
 - open/close 3680 (time 1788905051.38 total 30.01 last 123.99)
 - open/close 5124 (time 1788905061.39 total 40.02 last 144.32)
 - open/close 6580 (time 1788905071.39 total 50.02 last 145.56)
 - open/close 8886 (time 1788905081.39 total 60.02 last 230.54)
total: 10000 open/close in 63.71 seconds: 156.96 ops/second
 - unlinked 0 (time 1788905086 ; total 0 ; last 0)
total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg256-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg256-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.156@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg256-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1
Started lustre-OST0000
oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg256-client.virtnet:  -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre
On OST0, 19446 used inodes rc=0
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg256-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg256-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.156@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.156@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.156@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.156@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg256-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1
Started lustre-OST0000
oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg256-client.virtnet:  -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg256-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg256-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server
stop mds service on oleg256-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server

unloading modules via unload_modules_local on: 'oleg256-server'
oleg256-server: oleg256-server.virtnet: executing unload_modules_local
oleg256-server: 
modules unloaded.
pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2
pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2
pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2