== conf-sanity test 69: replace an OST with the same index ========================================================== 18:42:38 (1781217758)
start mds service on oleg330-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg330-server'
oleg330-server: oleg330-server.virtnet: executing load_modules_local
oleg330-server: Loading modules from /home/green/git/lustre-release/lustre
oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg330-server: detected 4 online CPUs by sysfs
oleg330-server: MODOPTS_LIBCFS=
oleg330-server: Force libcfs to create 2 CPU partitions
oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg330-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg330-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1
Started lustre-OST0000
oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg330-client.virtnet:  -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 388 (time 1781217853.18 total 10.00 last 38.79)
 - open/close 830 (time 1781217863.19 total 20.01 last 44.17)
 - open/close 1415 (time 1781217873.19 total 30.01 last 58.50)
 - open/close 1888 (time 1781217883.19 total 40.02 last 47.26)
 - open/close 2421 (time 1781217893.22 total 50.04 last 53.18)
 - open/close 2969 (time 1781217903.24 total 60.06 last 54.70)
 - open/close 3565 (time 1781217913.24 total 70.06 last 59.60)
 - open/close 4170 (time 1781217923.25 total 80.07 last 60.43)
 - open/close 4720 (time 1781217933.29 total 90.11 last 54.78)
 - open/close 5220 (time 1781217943.31 total 100.14 last 49.89)
 - open/close 5729 (time 1781217953.32 total 110.15 last 50.85)
 - open/close 6179 (time 1781217963.33 total 120.16 last 44.96)
 - open/close 6676 (time 1781217973.33 total 130.16 last 49.69)
 - open/close 7157 (time 1781217983.34 total 140.17 last 48.06)
 - open/close 7685 (time 1781217993.36 total 150.18 last 52.74)
 - open/close 8227 (time 1781218003.36 total 160.18 last 54.18)
 - open/close 8776 (time 1781218013.36 total 170.19 last 54.86)
 - open/close 9345 (time 1781218023.37 total 180.19 last 56.86)
 - open/close 9857 (time 1781218033.39 total 190.21 last 51.13)
total: 10000 open/close in 192.36 seconds: 51.99 ops/second
 - unlinked 0 (time 1781218040 ; total 0 ; last 0)
total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second
 - open/close 483 (time 1781218171.06 total 10.00 last 48.30)
 - open/close 961 (time 1781218181.08 total 20.02 last 47.70)
 - open/close 1460 (time 1781218191.09 total 30.04 last 49.83)
 - open/close 2011 (time 1781218201.10 total 40.05 last 55.04)
 - open/close 2587 (time 1781218211.11 total 50.06 last 57.54)
 - open/close 3124 (time 1781218221.14 total 60.08 last 53.55)
 - open/close 3651 (time 1781218231.15 total 70.09 last 52.65)
 - open/close 4200 (time 1781218241.15 total 80.09 last 54.89)
 - open/close 4760 (time 1781218251.15 total 90.10 last 55.97)
 - open/close 5313 (time 1781218261.16 total 100.11 last 55.26)
 - open/close 5950 (time 1781218271.17 total 110.11 last 63.67)
 - open/close 6499 (time 1781218281.18 total 120.13 last 54.82)
 - open/close 7070 (time 1781218291.19 total 130.14 last 57.03)
 - open/close 7645 (time 1781218301.20 total 140.15 last 57.45)
 - open/close 8167 (time 1781218311.22 total 150.17 last 52.09)
 - open/close 8730 (time 1781218321.23 total 160.17 last 56.26)
 - open/close 9211 (time 1781218331.23 total 170.18 last 48.10)
 - open/close 9795 (time 1781218341.24 total 180.18 last 58.37)
total: 10000 open/close in 183.74 seconds: 54.43 ops/second
 - unlinked 0 (time 1781218349 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 617 (time 1781218464.12 total 10.02 last 61.61)
 - open/close 1269 (time 1781218474.14 total 20.03 last 65.11)
 - open/close 1959 (time 1781218484.14 total 30.03 last 68.96)
 - open/close 2611 (time 1781218494.15 total 40.04 last 65.17)
 - open/close 3317 (time 1781218504.15 total 50.05 last 70.55)
 - open/close 3973 (time 1781218514.16 total 60.05 last 65.54)
 - open/close 4638 (time 1781218524.19 total 70.08 last 66.34)
 - open/close 5301 (time 1781218534.20 total 80.10 last 66.19)
 - open/close 5955 (time 1781218544.20 total 90.10 last 65.40)
 - open/close 6634 (time 1781218554.21 total 100.10 last 67.89)
 - open/close 7283 (time 1781218564.21 total 110.11 last 64.85)
 - open/close 7958 (time 1781218574.23 total 120.12 last 67.37)
 - open/close 8646 (time 1781218584.24 total 130.13 last 68.74)
 - open/close 9245 (time 1781218594.25 total 140.14 last 59.85)
 - open/close 9908 (time 1781218604.26 total 150.15 last 66.25)
total: 10000 open/close in 151.60 seconds: 65.96 ops/second
 - unlinked 0 (time 1781218610 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 660 (time 1781218719.99 total 10.01 last 65.95)
 - open/close 1359 (time 1781218730.00 total 20.02 last 69.85)
 - open/close 2019 (time 1781218740.01 total 30.03 last 65.90)
 - open/close 2721 (time 1781218750.02 total 40.03 last 70.17)
 - open/close 3439 (time 1781218760.02 total 50.04 last 71.76)
 - open/close 4160 (time 1781218770.03 total 60.04 last 72.07)
 - open/close 4816 (time 1781218780.03 total 70.05 last 65.58)
 - open/close 5450 (time 1781218790.04 total 80.05 last 63.37)
 - open/close 6081 (time 1781218800.06 total 90.07 last 62.97)
 - open/close 6513 (time 1781218810.06 total 100.08 last 43.17)
 - open/close 7061 (time 1781218820.08 total 110.09 last 54.73)
 - open/close 7603 (time 1781218830.10 total 120.12 last 54.08)
 - open/close 8164 (time 1781218840.12 total 130.13 last 56.01)
 - open/close 8772 (time 1781218850.12 total 140.14 last 60.77)
 - open/close 9413 (time 1781218860.13 total 150.14 last 64.06)
 - open/close 9948 (time 1781218870.14 total 160.15 last 53.45)
total: 10000 open/close in 160.99 seconds: 62.11 ops/second
 - unlinked 0 (time 1781218875 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 660 (time 1781218991.26 total 10.01 last 65.97)
 - open/close 1259 (time 1781219001.26 total 20.01 last 59.86)
 - open/close 1913 (time 1781219011.27 total 30.02 last 65.35)
 - open/close 2595 (time 1781219021.27 total 40.02 last 68.20)
 - open/close 3266 (time 1781219031.28 total 50.03 last 67.04)
 - open/close 3879 (time 1781219041.28 total 60.03 last 61.30)
 - open/close 4547 (time 1781219051.28 total 70.03 last 66.79)
 - open/close 5148 (time 1781219061.30 total 80.04 last 60.01)
 - open/close 5717 (time 1781219071.31 total 90.05 last 56.86)
 - open/close 6264 (time 1781219081.31 total 100.06 last 54.66)
 - open/close 6821 (time 1781219091.33 total 110.07 last 55.63)
 - open/close 7405 (time 1781219101.33 total 120.07 last 58.39)
 - open/close 7979 (time 1781219111.35 total 130.10 last 57.26)
 - open/close 8475 (time 1781219121.36 total 140.10 last 49.58)
 - open/close 9086 (time 1781219131.37 total 150.12 last 61.00)
 - open/close 9688 (time 1781219141.39 total 160.13 last 60.11)
total: 10000 open/close in 164.41 seconds: 60.82 ops/second
 - unlinked 0 (time 1781219150 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 486 (time 1781219263.47 total 10.01 last 48.55)
 - open/close 1021 (time 1781219273.48 total 20.02 last 53.43)
 - open/close 1633 (time 1781219283.49 total 30.03 last 61.14)
 - open/close 2208 (time 1781219293.49 total 40.03 last 57.49)
 - open/close 2798 (time 1781219303.53 total 50.07 last 58.81)
 - open/close 3343 (time 1781219313.53 total 60.07 last 54.46)
 - open/close 3955 (time 1781219323.54 total 70.08 last 61.16)
 - open/close 4633 (time 1781219333.55 total 80.09 last 67.74)
 - open/close 5319 (time 1781219343.56 total 90.10 last 68.52)
 - open/close 5926 (time 1781219353.57 total 100.11 last 60.67)
 - open/close 6579 (time 1781219363.57 total 110.11 last 65.30)
 - open/close 7179 (time 1781219373.57 total 120.11 last 59.96)
 - open/close 7774 (time 1781219383.58 total 130.12 last 59.48)
 - open/close 8385 (time 1781219393.58 total 140.12 last 61.07)
 - open/close 8912 (time 1781219403.59 total 150.13 last 52.64)
 - open/close 9497 (time 1781219413.62 total 160.16 last 58.32)
total: 10000 open/close in 168.69 seconds: 59.28 ops/second
 - unlinked 0 (time 1781219426 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 635 (time 1781219540.44 total 10.00 last 63.49)
 - open/close 1267 (time 1781219550.45 total 20.01 last 63.16)
 - open/close 1912 (time 1781219560.45 total 30.01 last 64.46)
 - open/close 2603 (time 1781219570.46 total 40.02 last 69.05)
 - open/close 3198 (time 1781219580.46 total 50.03 last 59.48)
 - open/close 3789 (time 1781219590.47 total 60.03 last 59.08)
 - open/close 4372 (time 1781219600.47 total 70.03 last 58.28)
 - open/close 5002 (time 1781219610.47 total 80.03 last 62.99)
 - open/close 5670 (time 1781219620.48 total 90.04 last 66.75)
 - open/close 6340 (time 1781219630.49 total 100.05 last 66.96)
 - open/close 6941 (time 1781219640.49 total 110.05 last 60.09)
 - open/close 7593 (time 1781219650.49 total 120.05 last 65.18)
 - open/close 8165 (time 1781219660.50 total 130.06 last 57.18)
 - open/close 8763 (time 1781219670.53 total 140.09 last 59.62)
 - open/close 9301 (time 1781219680.53 total 150.09 last 53.76)
 - open/close 9846 (time 1781219690.55 total 160.11 last 54.40)
total: 10000 open/close in 162.82 seconds: 61.42 ops/second
 - unlinked 0 (time 1781219699 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 680 (time 1781219813.28 total 10.01 last 67.92)
 - open/close 1351 (time 1781219823.29 total 20.02 last 67.04)
 - open/close 1989 (time 1781219833.30 total 30.03 last 63.72)
 - open/close 2657 (time 1781219843.31 total 40.04 last 66.78)
 - open/close 3360 (time 1781219853.32 total 50.05 last 70.22)
 - open/close 4061 (time 1781219863.33 total 60.06 last 70.01)
 - open/close 4660 (time 1781219873.33 total 70.06 last 59.88)
 - open/close 5248 (time 1781219883.34 total 80.07 last 58.76)
 - open/close 5918 (time 1781219893.35 total 90.08 last 66.93)
 - open/close 6599 (time 1781219903.36 total 100.09 last 68.05)
 - open/close 7227 (time 1781219913.36 total 110.10 last 62.77)
 - open/close 7901 (time 1781219923.37 total 120.10 last 67.35)
 - open/close 8563 (time 1781219933.37 total 130.11 last 66.18)
 - open/close 9222 (time 1781219943.39 total 140.12 last 65.79)
 - open/close 9810 (time 1781219953.41 total 150.14 last 58.67)
total: 10000 open/close in 153.24 seconds: 65.26 ops/second
 - unlinked 0 (time 1781219962 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 499 (time 1781220081.87 total 10.02 last 49.81)
 - open/close 1214 (time 1781220091.88 total 20.02 last 71.45)
 - open/close 1873 (time 1781220101.90 total 30.04 last 65.77)
 - open/close 2543 (time 1781220111.91 total 40.06 last 66.89)
 - open/close 3201 (time 1781220121.94 total 50.09 last 65.60)
 - open/close 3875 (time 1781220131.96 total 60.10 last 67.32)
 - open/close 4499 (time 1781220141.97 total 70.12 last 62.31)
 - open/close 5157 (time 1781220151.98 total 80.13 last 65.73)
 - open/close 5717 (time 1781220161.99 total 90.14 last 55.96)
 - open/close 6368 (time 1781220172.00 total 100.14 last 65.04)
 - open/close 7002 (time 1781220182.01 total 110.16 last 63.31)
 - open/close 7565 (time 1781220192.02 total 120.17 last 56.22)
 - open/close 8094 (time 1781220202.03 total 130.18 last 52.87)
 - open/close 8656 (time 1781220212.04 total 140.19 last 56.14)
 - open/close 9314 (time 1781220222.05 total 150.20 last 65.73)
 - open/close 9937 (time 1781220232.06 total 160.20 last 62.27)
total: 10000 open/close in 161.27 seconds: 62.01 ops/second
 - unlinked 0 (time 1781220237 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 688 (time 1781220352.53 total 10.04 last 68.54)
 - open/close 1339 (time 1781220362.54 total 20.05 last 65.05)
 - open/close 1993 (time 1781220372.57 total 30.08 last 65.19)
 - open/close 2642 (time 1781220382.58 total 40.08 last 64.88)
 - open/close 3292 (time 1781220392.59 total 50.09 last 64.93)
 - open/close 3954 (time 1781220402.59 total 60.09 last 66.18)
 - open/close 4620 (time 1781220412.62 total 70.12 last 66.42)
 - open/close 5247 (time 1781220422.63 total 80.13 last 62.63)
 - open/close 5806 (time 1781220432.64 total 90.14 last 55.83)
 - open/close 6507 (time 1781220442.66 total 100.16 last 69.97)
 - open/close 7192 (time 1781220452.66 total 110.17 last 68.45)
 - open/close 7889 (time 1781220462.68 total 120.18 last 69.62)
 - open/close 8543 (time 1781220472.68 total 130.18 last 65.39)
 - open/close 9128 (time 1781220482.69 total 140.19 last 58.45)
 - open/close 9789 (time 1781220492.69 total 150.20 last 66.05)
total: 10000 open/close in 153.12 seconds: 65.31 ops/second
 - unlinked 0 (time 1781220500 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg330-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg330-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.130@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg330-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1
Started lustre-OST0000
oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg330-client.virtnet:  -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg330-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg330-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.130@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.130@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.130@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg330-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1
Started lustre-OST0000
oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg330-client.virtnet:  -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg330-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg330-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server
stop mds service on oleg330-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server

unloading modules via unload_modules_local on: 'oleg330-server'
oleg330-server: oleg330-server.virtnet: executing unload_modules_local
oleg330-server: 
modules unloaded.
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2
pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2
pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2
pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2