== conf-sanity test 69: replace an OST with the same index ========================================================== 17:59:45 (1784584785)
start mds service on oleg260-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg260-server'
oleg260-server: oleg260-server.virtnet: executing load_modules_local
oleg260-server: Loading modules from /home/green/git/lustre-release/lustre
oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg260-server: detected 4 online CPUs by sysfs
oleg260-server: MODOPTS_LIBCFS=
oleg260-server: Force libcfs to create 2 CPU partitions
oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg260-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-OST0000
oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg260-client.virtnet:  -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 616 (time 1784584871.74 total 10.02 last 61.48)
 - open/close 1342 (time 1784584881.74 total 20.02 last 72.56)
 - open/close 2140 (time 1784584891.75 total 30.03 last 79.74)
 - open/close 3036 (time 1784584901.76 total 40.04 last 89.53)
 - open/close 3794 (time 1784584911.78 total 50.07 last 75.61)
 - open/close 4698 (time 1784584921.79 total 60.07 last 90.34)
 - open/close 5460 (time 1784584931.80 total 70.09 last 76.09)
 - open/close 6125 (time 1784584941.81 total 80.09 last 66.46)
 - open/close 6829 (time 1784584951.81 total 90.10 last 70.37)
 - open/close 7530 (time 1784584961.82 total 100.10 last 70.08)
 - open/close 8139 (time 1784584971.82 total 110.10 last 60.89)
 - open/close 8789 (time 1784584981.85 total 120.13 last 64.80)
 - open/close 9456 (time 1784584991.86 total 130.14 last 66.64)
total: 10000 open/close in 137.87 seconds: 72.53 ops/second
 - unlinked 0 (time 1784585003 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 730 (time 1784585103.94 total 10.01 last 72.92)
 - open/close 1451 (time 1784585113.95 total 20.02 last 72.03)
 - open/close 2144 (time 1784585123.96 total 30.03 last 69.24)
 - open/close 2854 (time 1784585133.98 total 40.05 last 70.87)
 - open/close 3628 (time 1784585143.98 total 50.05 last 77.37)
 - open/close 4242 (time 1784585154.01 total 60.08 last 61.25)
 - open/close 4882 (time 1784585164.02 total 70.09 last 63.94)
 - open/close 5501 (time 1784585174.03 total 80.10 last 61.81)
 - open/close 6196 (time 1784585184.04 total 90.11 last 69.45)
 - open/close 6915 (time 1784585194.06 total 100.13 last 71.77)
 - open/close 7654 (time 1784585204.07 total 110.14 last 73.82)
 - open/close 8396 (time 1784585214.07 total 120.14 last 74.19)
 - open/close 9105 (time 1784585224.09 total 130.16 last 70.77)
 - open/close 9786 (time 1784585234.10 total 140.17 last 68.02)
total: 10000 open/close in 142.94 seconds: 69.96 ops/second
 - unlinked 0 (time 1784585241 ; total 1 ; last 1)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 531 (time 1784585345.58 total 10.01 last 53.03)
 - open/close 1161 (time 1784585355.60 total 20.02 last 62.93)
 - open/close 1713 (time 1784585365.61 total 30.03 last 55.14)
 - open/close 2351 (time 1784585375.61 total 40.04 last 63.77)
 - open/close 2992 (time 1784585385.61 total 50.04 last 64.09)
 - open/close 3673 (time 1784585395.61 total 60.04 last 68.08)
 - open/close 4340 (time 1784585405.62 total 70.04 last 66.69)
 - open/close 5013 (time 1784585415.62 total 80.04 last 67.30)
 - open/close 5732 (time 1784585425.62 total 90.05 last 71.86)
 - open/close 6415 (time 1784585435.63 total 100.06 last 68.22)
 - open/close 7165 (time 1784585445.64 total 110.07 last 74.95)
 - open/close 7828 (time 1784585455.66 total 120.08 last 66.19)
 - open/close 8564 (time 1784585465.67 total 130.09 last 73.53)
 - open/close 9391 (time 1784585475.67 total 140.10 last 82.64)
total: 10000 open/close in 147.65 seconds: 67.73 ops/second
 - unlinked 0 (time 1784585488 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 824 (time 1784585596.12 total 10.01 last 82.32)
 - open/close 1510 (time 1784585606.13 total 20.02 last 68.52)
 - open/close 2244 (time 1784585616.13 total 30.02 last 73.39)
 - open/close 2899 (time 1784585626.13 total 40.02 last 65.49)
 - open/close 3685 (time 1784585636.13 total 50.03 last 78.57)
 - open/close 4314 (time 1784585646.15 total 60.04 last 62.80)
 - open/close 4897 (time 1784585656.16 total 70.05 last 58.26)
 - open/close 5491 (time 1784585666.16 total 80.05 last 59.37)
 - open/close 6256 (time 1784585676.16 total 90.06 last 76.48)
 - open/close 6949 (time 1784585686.17 total 100.07 last 69.24)
 - open/close 7600 (time 1784585696.18 total 110.07 last 65.04)
 - open/close 8379 (time 1784585706.19 total 120.08 last 77.83)
 - open/close 9145 (time 1784585716.21 total 130.11 last 76.43)
 - open/close 9881 (time 1784585726.22 total 140.11 last 73.56)
total: 10000 open/close in 141.63 seconds: 70.61 ops/second
 - unlinked 0 (time 1784585732 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 697 (time 1784585836.58 total 10.04 last 69.40)
 - open/close 1410 (time 1784585846.60 total 20.06 last 71.19)
 - open/close 2161 (time 1784585856.61 total 30.07 last 75.04)
 - open/close 2954 (time 1784585866.61 total 40.07 last 79.24)
 - open/close 3689 (time 1784585876.63 total 50.09 last 73.38)
 - open/close 4386 (time 1784585886.63 total 60.10 last 69.66)
 - open/close 4965 (time 1784585896.64 total 70.10 last 57.89)
 - open/close 5640 (time 1784585906.65 total 80.11 last 67.44)
 - open/close 6342 (time 1784585916.65 total 90.11 last 70.19)
 - open/close 7090 (time 1784585926.65 total 100.11 last 74.80)
 - open/close 7770 (time 1784585936.66 total 110.12 last 67.95)
 - open/close 8504 (time 1784585946.66 total 120.12 last 73.38)
 - open/close 9270 (time 1784585956.67 total 130.13 last 76.51)
 - open/close 9956 (time 1784585966.67 total 140.14 last 68.56)
total: 10000 open/close in 140.83 seconds: 71.01 ops/second
 - unlinked 0 (time 1784585972 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 698 (time 1784586071.68 total 10.01 last 69.72)
 - open/close 1468 (time 1784586081.68 total 20.01 last 76.98)
 - open/close 2226 (time 1784586091.70 total 30.03 last 75.68)
 - open/close 2981 (time 1784586101.70 total 40.03 last 75.48)
 - open/close 3660 (time 1784586111.70 total 50.03 last 67.90)
 - open/close 4267 (time 1784586121.70 total 60.04 last 60.68)
 - open/close 4873 (time 1784586131.71 total 70.04 last 60.59)
 - open/close 5534 (time 1784586141.72 total 80.05 last 66.05)
 - open/close 6288 (time 1784586151.72 total 90.05 last 75.40)
 - open/close 7036 (time 1784586161.72 total 100.05 last 74.79)
 - open/close 7777 (time 1784586171.73 total 110.06 last 74.04)
 - open/close 8471 (time 1784586181.74 total 120.07 last 69.33)
 - open/close 9209 (time 1784586191.74 total 130.07 last 73.76)
 - open/close 9864 (time 1784586201.74 total 140.07 last 65.50)
total: 10000 open/close in 142.08 seconds: 70.38 ops/second
 - unlinked 0 (time 1784586208 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 689 (time 1784586312.90 total 10.01 last 68.84)
 - open/close 1419 (time 1784586322.91 total 20.02 last 72.95)
 - open/close 1958 (time 1784586332.91 total 30.02 last 53.85)
 - open/close 2649 (time 1784586342.92 total 40.03 last 69.05)
 - open/close 3353 (time 1784586352.95 total 50.06 last 70.23)
 - open/close 4102 (time 1784586362.96 total 60.07 last 74.82)
 - open/close 4905 (time 1784586372.96 total 70.07 last 80.24)
 - open/close 5578 (time 1784586382.99 total 80.10 last 67.14)
 - open/close 6388 (time 1784586392.99 total 90.10 last 80.95)
 - open/close 7200 (time 1784586403.00 total 100.11 last 81.14)
 - open/close 7828 (time 1784586413.01 total 110.12 last 62.74)
 - open/close 8640 (time 1784586423.02 total 120.13 last 81.09)
 - open/close 9313 (time 1784586433.03 total 130.14 last 67.28)
 - open/close 9935 (time 1784586443.04 total 140.15 last 62.14)
total: 10000 open/close in 141.01 seconds: 70.92 ops/second
 - unlinked 0 (time 1784586448 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 595 (time 1784586560.50 total 10.01 last 59.46)
 - open/close 1137 (time 1784586570.52 total 20.02 last 54.14)
 - open/close 1705 (time 1784586580.52 total 30.02 last 56.79)
 - open/close 2295 (time 1784586590.53 total 40.03 last 58.93)
 - open/close 3001 (time 1784586600.54 total 50.04 last 70.53)
 - open/close 3670 (time 1784586610.55 total 60.05 last 66.84)
 - open/close 4484 (time 1784586620.55 total 70.05 last 81.38)
 - open/close 5184 (time 1784586630.58 total 80.08 last 69.79)
 - open/close 5905 (time 1784586640.59 total 90.09 last 72.05)
 - open/close 6626 (time 1784586650.60 total 100.10 last 72.02)
 - open/close 7388 (time 1784586660.61 total 110.11 last 76.13)
 - open/close 8005 (time 1784586670.62 total 120.12 last 61.64)
 - open/close 8738 (time 1784586680.62 total 130.13 last 73.25)
 - open/close 9444 (time 1784586690.63 total 140.14 last 70.53)
total: 10000 open/close in 147.49 seconds: 67.80 ops/second
 - unlinked 0 (time 1784586702 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 696 (time 1784586815.26 total 10.01 last 69.50)
 - open/close 1320 (time 1784586825.27 total 20.02 last 62.36)
 - open/close 2138 (time 1784586835.29 total 30.05 last 81.57)
 - open/close 2986 (time 1784586845.30 total 40.06 last 84.73)
 - open/close 3696 (time 1784586855.31 total 50.06 last 70.94)
 - open/close 4404 (time 1784586865.31 total 60.07 last 70.79)
 - open/close 5088 (time 1784586875.31 total 70.07 last 68.39)
 - open/close 5845 (time 1784586885.31 total 80.07 last 75.69)
 - open/close 6597 (time 1784586895.33 total 90.08 last 75.12)
 - open/close 7298 (time 1784586905.33 total 100.09 last 70.05)
 - open/close 7833 (time 1784586915.34 total 110.09 last 53.48)
 - open/close 8185 (time 1784586925.34 total 120.09 last 35.18)
 - open/close 8696 (time 1784586935.37 total 130.12 last 50.97)
 - open/close 9298 (time 1784586945.37 total 140.12 last 60.18)
total: 10000 open/close in 150.02 seconds: 66.66 ops/second
 - unlinked 0 (time 1784586961 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 666 (time 1784587067.13 total 10.01 last 66.52)
 - open/close 1381 (time 1784587077.13 total 20.02 last 71.47)
 - open/close 2154 (time 1784587087.14 total 30.02 last 77.28)
 - open/close 2897 (time 1784587097.14 total 40.02 last 74.25)
 - open/close 3661 (time 1784587107.17 total 50.05 last 76.22)
 - open/close 4448 (time 1784587117.17 total 60.05 last 78.69)
 - open/close 5228 (time 1784587127.18 total 70.06 last 77.92)
 - open/close 5900 (time 1784587137.18 total 80.07 last 67.16)
 - open/close 6638 (time 1784587147.19 total 90.07 last 73.74)
 - open/close 7442 (time 1784587157.20 total 100.09 last 80.30)
 - open/close 8218 (time 1784587167.21 total 110.09 last 77.57)
 - open/close 8911 (time 1784587177.22 total 120.10 last 69.24)
 - open/close 9682 (time 1784587187.23 total 130.11 last 77.03)
total: 10000 open/close in 133.89 seconds: 74.69 ops/second
 - unlinked 0 (time 1784587195 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg260-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg260-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg260-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-OST0000
oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg260-client.virtnet:  -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre
On OST0, 11414 used inodes rc=0
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg260-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg260-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.160@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.160@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg260-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-OST0000
oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg260-client.virtnet:  -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg260-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg260-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server
stop mds service on oleg260-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server

unloading modules via unload_modules_local on: 'oleg260-server'
oleg260-server: oleg260-server.virtnet: executing unload_modules_local
oleg260-server: 
modules unloaded.
pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2
pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2