== conf-sanity test 69: replace an OST with the same index ========================================================== 11:07:44 (1781622464)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 763 (time 1781622551.16 total 10.01 last 76.21)
 - open/close 1551 (time 1781622561.18 total 20.03 last 78.62)
 - open/close 2234 (time 1781622571.19 total 30.05 last 68.22)
 - open/close 2974 (time 1781622581.20 total 40.05 last 73.94)
 - open/close 3869 (time 1781622591.21 total 50.06 last 89.45)
 - open/close 4682 (time 1781622601.22 total 60.07 last 81.19)
 - open/close 5578 (time 1781622611.24 total 70.09 last 89.43)
 - open/close 6416 (time 1781622621.24 total 80.10 last 83.75)
 - open/close 7353 (time 1781622631.25 total 90.10 last 93.66)
 - open/close 8266 (time 1781622641.25 total 100.11 last 91.25)
 - open/close 9101 (time 1781622651.26 total 110.11 last 83.48)
total: 10000 open/close in 119.97 seconds: 83.36 ops/second
 - unlinked 0 (time 1781622665 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 867 (time 1781622757.17 total 10.01 last 86.64)
 - open/close 1763 (time 1781622767.17 total 20.01 last 89.53)
 - open/close 2682 (time 1781622777.20 total 30.04 last 91.67)
 - open/close 3324 (time 1781622787.21 total 40.05 last 64.12)
 - open/close 4036 (time 1781622797.22 total 50.06 last 71.13)
 - open/close 4756 (time 1781622807.22 total 60.06 last 72.00)
 - open/close 5526 (time 1781622817.23 total 70.07 last 76.95)
 - open/close 6404 (time 1781622827.24 total 80.08 last 87.71)
 - open/close 7199 (time 1781622837.24 total 90.08 last 79.47)
 - open/close 8010 (time 1781622847.24 total 100.09 last 81.08)
 - open/close 8849 (time 1781622857.25 total 110.09 last 83.85)
 - open/close 9663 (time 1781622867.25 total 120.10 last 81.37)
total: 10000 open/close in 123.86 seconds: 80.74 ops/second
 - unlinked 0 (time 1781622874 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 767 (time 1781622964.29 total 10.01 last 76.64)
 - open/close 1518 (time 1781622974.29 total 20.02 last 75.04)
 - open/close 2400 (time 1781622984.32 total 30.04 last 88.00)
 - open/close 3243 (time 1781622994.32 total 40.04 last 84.28)
 - open/close 4148 (time 1781623004.33 total 50.05 last 90.42)
 - open/close 5051 (time 1781623014.33 total 60.05 last 90.27)
 - open/close 5936 (time 1781623024.33 total 70.05 last 88.48)
 - open/close 6831 (time 1781623034.34 total 80.06 last 89.41)
 - open/close 7677 (time 1781623044.34 total 90.07 last 84.59)
 - open/close 8500 (time 1781623054.35 total 100.07 last 82.27)
 - open/close 9352 (time 1781623064.36 total 110.08 last 85.14)
total: 10000 open/close in 117.70 seconds: 84.96 ops/second
 - unlinked 0 (time 1781623075 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 913 (time 1781623162.16 total 10.00 last 91.28)
 - open/close 1787 (time 1781623172.16 total 20.01 last 87.34)
 - open/close 2635 (time 1781623182.17 total 30.01 last 84.78)
 - open/close 3541 (time 1781623192.17 total 40.02 last 90.54)
 - open/close 4514 (time 1781623202.18 total 50.02 last 97.27)
 - open/close 5347 (time 1781623212.19 total 60.03 last 83.19)
 - open/close 6106 (time 1781623222.20 total 70.04 last 75.83)
 - open/close 6966 (time 1781623232.20 total 80.04 last 85.99)
 - open/close 7832 (time 1781623242.22 total 90.06 last 86.41)
 - open/close 8775 (time 1781623252.22 total 100.07 last 94.28)
 - open/close 9694 (time 1781623262.23 total 110.08 last 91.83)
total: 10000 open/close in 114.29 seconds: 87.50 ops/second
 - unlinked 0 (time 1781623270 ; total 0 ; last 0)
total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second
 - open/close 993 (time 1781623350.17 total 10.00 last 99.27)
 - open/close 1835 (time 1781623360.17 total 20.00 last 84.19)
 - open/close 2642 (time 1781623370.18 total 30.02 last 80.59)
 - open/close 3493 (time 1781623380.19 total 40.02 last 85.08)
 - open/close 4332 (time 1781623390.19 total 50.02 last 83.89)
 - open/close 5318 (time 1781623400.19 total 60.03 last 98.53)
 - open/close 6312 (time 1781623410.20 total 70.04 last 99.34)
 - open/close 7376 (time 1781623420.20 total 80.04 last 106.38)
 - open/close 8197 (time 1781623430.21 total 90.04 last 82.08)
 - open/close 9050 (time 1781623440.21 total 100.04 last 85.29)
 - open/close 9908 (time 1781623450.21 total 110.05 last 85.77)
total: 10000 open/close in 110.99 seconds: 90.10 ops/second
 - unlinked 0 (time 1781623454 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 916 (time 1781623535.07 total 10.00 last 91.58)
 - open/close 1728 (time 1781623545.08 total 20.01 last 81.11)
 - open/close 2635 (time 1781623555.08 total 30.02 last 90.66)
 - open/close 3566 (time 1781623565.11 total 40.04 last 92.88)
 - open/close 4477 (time 1781623575.11 total 50.05 last 91.04)
 - open/close 5289 (time 1781623585.12 total 60.06 last 81.16)
 - open/close 6177 (time 1781623595.12 total 70.06 last 88.76)
 - open/close 7041 (time 1781623605.13 total 80.06 last 86.36)
 - open/close 7934 (time 1781623615.13 total 90.07 last 89.27)
 - open/close 8905 (time 1781623625.13 total 100.07 last 97.07)
 - open/close 9909 (time 1781623635.13 total 110.07 last 100.39)
total: 10000 open/close in 110.87 seconds: 90.20 ops/second
 - unlinked 0 (time 1781623639 ; total 0 ; last 0)
total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second
 - open/close 876 (time 1781623717.32 total 10.01 last 87.49)
 - open/close 1710 (time 1781623727.33 total 20.02 last 83.34)
 - open/close 2673 (time 1781623737.33 total 30.02 last 96.28)
 - open/close 3729 (time 1781623747.34 total 40.03 last 105.55)
 - open/close 4516 (time 1781623757.34 total 50.03 last 78.64)
 - open/close 5509 (time 1781623767.35 total 60.04 last 99.30)
 - open/close 6347 (time 1781623777.35 total 70.04 last 83.79)
 - open/close 7262 (time 1781623787.35 total 80.04 last 91.44)
 - open/close 8229 (time 1781623797.36 total 90.05 last 96.62)
 - open/close 9128 (time 1781623807.37 total 100.06 last 89.81)
 - open/close 9967 (time 1781623817.38 total 110.07 last 83.81)
total: 10000 open/close in 110.39 seconds: 90.59 ops/second
 - unlinked 0 (time 1781623820 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 953 (time 1781623900.12 total 10.00 last 95.26)
 - open/close 1948 (time 1781623910.12 total 20.00 last 99.50)
 - open/close 2910 (time 1781623920.12 total 30.01 last 96.12)
 - open/close 3835 (time 1781623930.14 total 40.03 last 92.38)
 - open/close 4648 (time 1781623940.14 total 50.03 last 81.29)
 - open/close 5551 (time 1781623950.15 total 60.03 last 90.23)
 - open/close 6552 (time 1781623960.15 total 70.04 last 100.05)
 - open/close 7580 (time 1781623970.16 total 80.05 last 102.72)
 - open/close 8440 (time 1781623980.18 total 90.06 last 85.85)
 - open/close 9222 (time 1781623990.18 total 100.07 last 78.18)
total: 10000 open/close in 109.51 seconds: 91.31 ops/second
 - unlinked 0 (time 1781624002 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 991 (time 1781624078.34 total 10.00 last 99.06)
 - open/close 1879 (time 1781624088.34 total 20.01 last 88.76)
 - open/close 2679 (time 1781624098.35 total 30.01 last 79.99)
 - open/close 3599 (time 1781624108.35 total 40.01 last 91.99)
 - open/close 4474 (time 1781624118.35 total 50.01 last 87.48)
 - open/close 5354 (time 1781624128.35 total 60.02 last 87.95)
 - open/close 6249 (time 1781624138.36 total 70.03 last 89.42)
 - open/close 7090 (time 1781624148.37 total 80.04 last 84.01)
 - open/close 7920 (time 1781624158.38 total 90.05 last 82.91)
 - open/close 8754 (time 1781624168.39 total 100.05 last 83.37)
 - open/close 9651 (time 1781624178.40 total 110.07 last 89.58)
total: 10000 open/close in 113.52 seconds: 88.09 ops/second
 - unlinked 0 (time 1781624184 ; total 0 ; last 0)
total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second
 - open/close 898 (time 1781624263.59 total 10.01 last 89.73)
 - open/close 1882 (time 1781624273.59 total 20.02 last 98.32)
 - open/close 2744 (time 1781624283.59 total 30.02 last 86.19)
 - open/close 3639 (time 1781624293.60 total 40.02 last 89.49)
 - open/close 4547 (time 1781624303.60 total 50.03 last 90.72)
 - open/close 5459 (time 1781624313.61 total 60.04 last 91.13)
 - open/close 6245 (time 1781624323.61 total 70.04 last 78.59)
 - open/close 7034 (time 1781624333.62 total 80.04 last 78.86)
 - open/close 7921 (time 1781624343.63 total 90.06 last 88.57)
 - open/close 8874 (time 1781624353.65 total 100.08 last 95.13)
 - open/close 9870 (time 1781624363.66 total 110.08 last 99.53)
total: 10000 open/close in 111.44 seconds: 89.73 ops/second
 - unlinked 0 (time 1781624368 ; total 0 ; last 0)
total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
On OST0, 14222 used inodes rc=0
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2