== conf-sanity test 69: replace an OST with the same index ========================================================== 06:00:26 (1787997626)
start mds service on oleg131-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg131-server'
oleg131-server: oleg131-server.virtnet: executing load_modules_local
oleg131-server: Loading modules from /home/green/git/lustre-release/lustre
oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg131-server: detected 4 online CPUs by sysfs
oleg131-server: MODOPTS_LIBCFS=
oleg131-server: Force libcfs to create 2 CPU partitions
oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1003 (time 1787997726.09 total 10.00 last 100.29)
 - open/close 1811 (time 1787997736.10 total 20.01 last 80.75)
 - open/close 2769 (time 1787997746.11 total 30.01 last 95.75)
 - open/close 3511 (time 1787997756.11 total 40.02 last 74.17)
 - open/close 4356 (time 1787997766.13 total 50.03 last 84.34)
 - open/close 5002 (time 1787997776.14 total 60.05 last 64.53)
 - open/close 5931 (time 1787997786.14 total 70.05 last 92.85)
 - open/close 6788 (time 1787997796.15 total 80.06 last 85.63)
 - open/close 7566 (time 1787997806.19 total 90.10 last 77.49)
 - open/close 8383 (time 1787997816.20 total 100.10 last 81.66)
 - open/close 9185 (time 1787997826.21 total 110.12 last 80.10)
total: 10000 open/close in 119.24 seconds: 83.87 ops/second
 - unlinked 0 (time 1787997839 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 857 (time 1787997928.85 total 10.00 last 85.66)
 - open/close 1701 (time 1787997938.86 total 20.01 last 84.35)
 - open/close 2522 (time 1787997948.87 total 30.02 last 82.06)
 - open/close 3282 (time 1787997958.88 total 40.03 last 75.88)
 - open/close 4157 (time 1787997968.89 total 50.04 last 87.44)
 - open/close 4942 (time 1787997978.89 total 60.04 last 78.45)
 - open/close 5715 (time 1787997988.92 total 70.07 last 77.14)
 - open/close 6509 (time 1787997998.92 total 80.07 last 79.39)
 - open/close 7297 (time 1787998008.92 total 90.07 last 78.80)
 - open/close 8212 (time 1787998018.93 total 100.08 last 91.37)
 - open/close 9069 (time 1787998028.95 total 110.10 last 85.55)
 - open/close 9668 (time 1787998038.96 total 120.11 last 59.85)
total: 10000 open/close in 124.31 seconds: 80.44 ops/second
 - unlinked 0 (time 1787998047 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 721 (time 1787998143.60 total 10.02 last 71.99)
 - open/close 1470 (time 1787998153.61 total 20.02 last 74.84)
 - open/close 2435 (time 1787998163.61 total 30.02 last 96.49)
 - open/close 3231 (time 1787998173.62 total 40.03 last 79.52)
 - open/close 4053 (time 1787998183.63 total 50.04 last 82.15)
 - open/close 4785 (time 1787998193.63 total 60.04 last 73.19)
 - open/close 5652 (time 1787998203.63 total 70.04 last 86.68)
 - open/close 6538 (time 1787998213.64 total 80.05 last 88.51)
 - open/close 7382 (time 1787998223.64 total 90.06 last 84.38)
 - open/close 8090 (time 1787998233.65 total 100.06 last 70.76)
 - open/close 8777 (time 1787998243.69 total 110.10 last 68.43)
 - open/close 9552 (time 1787998253.69 total 120.11 last 77.47)
total: 10000 open/close in 124.91 seconds: 80.06 ops/second
 - unlinked 0 (time 1787998262 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 785 (time 1787998351.55 total 10.00 last 78.48)
 - open/close 1622 (time 1787998361.59 total 20.04 last 83.38)
 - open/close 2372 (time 1787998371.59 total 30.05 last 74.95)
 - open/close 3106 (time 1787998381.59 total 40.05 last 73.40)
 - open/close 3950 (time 1787998391.60 total 50.06 last 84.33)
 - open/close 4797 (time 1787998401.63 total 60.09 last 84.45)
 - open/close 5586 (time 1787998411.63 total 70.09 last 78.88)
 - open/close 6243 (time 1787998421.64 total 80.09 last 65.67)
 - open/close 6981 (time 1787998431.65 total 90.11 last 73.70)
 - open/close 7774 (time 1787998441.66 total 100.12 last 79.23)
 - open/close 8619 (time 1787998451.67 total 110.12 last 84.45)
 - open/close 9437 (time 1787998461.68 total 120.13 last 81.73)
total: 10000 open/close in 128.12 seconds: 78.05 ops/second
 - unlinked 0 (time 1787998473 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 845 (time 1787998568.73 total 10.01 last 84.44)
 - open/close 1657 (time 1787998578.76 total 20.03 last 80.99)
 - open/close 2383 (time 1787998588.77 total 30.04 last 72.51)
 - open/close 3224 (time 1787998598.78 total 40.06 last 83.99)
 - open/close 3878 (time 1787998608.80 total 50.07 last 65.30)
 - open/close 4665 (time 1787998618.80 total 60.08 last 78.67)
 - open/close 5567 (time 1787998628.80 total 70.08 last 90.17)
 - open/close 6427 (time 1787998638.81 total 80.09 last 85.93)
 - open/close 7230 (time 1787998648.82 total 90.10 last 80.24)
 - open/close 7964 (time 1787998658.82 total 100.10 last 73.39)
 - open/close 8676 (time 1787998668.82 total 110.10 last 71.18)
 - open/close 9593 (time 1787998678.83 total 120.11 last 91.63)
total: 10000 open/close in 125.94 seconds: 79.40 ops/second
 - unlinked 0 (time 1787998689 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 515 (time 1787998779.87 total 10.01 last 51.44)
 - open/close 1193 (time 1787998789.87 total 20.02 last 67.78)
 - open/close 1967 (time 1787998799.88 total 30.03 last 77.32)
 - open/close 2820 (time 1787998809.88 total 40.03 last 85.27)
 - open/close 3677 (time 1787998819.88 total 50.03 last 85.69)
 - open/close 4494 (time 1787998829.88 total 60.03 last 81.70)
 - open/close 5186 (time 1787998839.90 total 70.05 last 69.07)
 - open/close 5923 (time 1787998849.90 total 80.05 last 73.69)
 - open/close 6714 (time 1787998859.92 total 90.06 last 79.01)
 - open/close 7586 (time 1787998869.92 total 100.07 last 87.15)
 - open/close 8458 (time 1787998879.92 total 110.07 last 87.18)
 - open/close 9285 (time 1787998889.93 total 120.07 last 82.68)
total: 10000 open/close in 127.46 seconds: 78.45 ops/second
 - unlinked 0 (time 1787998901 ; total 0 ; last 0)
total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second
 - open/close 1115 (time 1787998984.73 total 10.00 last 111.45)
 - open/close 2749 (time 1787998994.73 total 20.01 last 163.28)
 - open/close 3691 (time 1787999004.75 total 30.03 last 94.06)
 - open/close 4625 (time 1787999014.78 total 40.05 last 93.15)
 - open/close 5240 (time 1787999024.79 total 50.07 last 61.42)
 - open/close 5840 (time 1787999034.80 total 60.08 last 59.91)
 - open/close 6587 (time 1787999044.82 total 70.10 last 74.57)
 - open/close 7240 (time 1787999054.82 total 80.10 last 65.29)
 - open/close 7852 (time 1787999064.84 total 90.12 last 61.09)
 - open/close 8585 (time 1787999074.85 total 100.12 last 73.29)
 - open/close 9377 (time 1787999084.86 total 110.14 last 79.07)
total: 10000 open/close in 117.73 seconds: 84.94 ops/second
 - unlinked 0 (time 1787999096 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 823 (time 1787999187.58 total 10.00 last 82.27)
 - open/close 1632 (time 1787999197.59 total 20.01 last 80.86)
 - open/close 2451 (time 1787999207.59 total 30.01 last 81.88)
 - open/close 3063 (time 1787999217.60 total 40.02 last 61.15)
 - open/close 3800 (time 1787999227.60 total 50.02 last 73.67)
 - open/close 4591 (time 1787999237.61 total 60.03 last 79.04)
 - open/close 5378 (time 1787999247.62 total 70.04 last 78.61)
 - open/close 6260 (time 1787999257.62 total 80.04 last 88.18)
 - open/close 7227 (time 1787999267.62 total 90.05 last 96.67)
 - open/close 8055 (time 1787999277.64 total 100.06 last 82.68)
 - open/close 8843 (time 1787999287.64 total 110.06 last 78.79)
 - open/close 9624 (time 1787999297.66 total 120.08 last 77.94)
total: 10000 open/close in 126.08 seconds: 79.32 ops/second
 - unlinked 0 (time 1787999308 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 973 (time 1787999399.48 total 10.01 last 97.23)
 - open/close 1898 (time 1787999409.49 total 20.01 last 92.44)
 - open/close 2743 (time 1787999419.49 total 30.01 last 84.49)
 - open/close 3804 (time 1787999429.49 total 40.02 last 106.08)
 - open/close 4689 (time 1787999439.50 total 50.02 last 88.45)
 - open/close 5351 (time 1787999449.52 total 60.04 last 66.08)
 - open/close 6146 (time 1787999459.52 total 70.04 last 79.49)
 - open/close 6994 (time 1787999469.52 total 80.05 last 84.76)
 - open/close 7639 (time 1787999479.53 total 90.06 last 64.41)
 - open/close 8539 (time 1787999489.55 total 100.07 last 89.90)
 - open/close 9295 (time 1787999499.55 total 110.08 last 75.55)
total: 10000 open/close in 118.77 seconds: 84.19 ops/second
 - unlinked 0 (time 1787999512 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 826 (time 1787999600.80 total 10.00 last 82.56)
 - open/close 1683 (time 1787999610.80 total 20.01 last 85.68)
 - open/close 2436 (time 1787999620.81 total 30.02 last 75.19)
 - open/close 3342 (time 1787999630.82 total 40.03 last 90.56)
 - open/close 4104 (time 1787999640.82 total 50.03 last 76.16)
 - open/close 4976 (time 1787999650.83 total 60.04 last 87.15)
 - open/close 5974 (time 1787999660.83 total 70.04 last 99.78)
 - open/close 7030 (time 1787999670.86 total 80.07 last 105.28)
 - open/close 7941 (time 1787999680.86 total 90.07 last 91.08)
 - open/close 8802 (time 1787999690.87 total 100.08 last 86.02)
 - open/close 9675 (time 1787999700.88 total 110.09 last 87.22)
total: 10000 open/close in 114.16 seconds: 87.59 ops/second
 - unlinked 0 (time 1787999709 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.131@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.131@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.131@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.131@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.131@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg131-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1
Started lustre-OST0000
oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg131-client.virtnet:  -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg131-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg131-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server
stop mds service on oleg131-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server

unloading modules via unload_modules_local on: 'oleg131-server'
oleg131-server: oleg131-server.virtnet: executing unload_modules_local
oleg131-server: 
modules unloaded.
pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2
pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2