== conf-sanity test 69: replace an OST with the same index ========================================================== 14:08:22 (1782756502)
start mds service on oleg417-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg417-server'
oleg417-server: oleg417-server.virtnet: executing load_modules_local
oleg417-server: Loading modules from /home/green/git/lustre-release/lustre
oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg417-server: detected 4 online CPUs by sysfs
oleg417-server: MODOPTS_LIBCFS=
oleg417-server: Force libcfs to create 2 CPU partitions
oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg417-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg417-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Started lustre-OST0000
oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg417-client.virtnet:  -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 583 (time 1782756596.18 total 10.04 last 58.09)
 - open/close 1211 (time 1782756606.19 total 20.04 last 62.75)
 - open/close 1790 (time 1782756616.20 total 30.06 last 57.82)
 - open/close 2425 (time 1782756626.21 total 40.06 last 63.47)
 - open/close 3080 (time 1782756636.22 total 50.08 last 65.40)
 - open/close 3749 (time 1782756646.23 total 60.09 last 66.83)
 - open/close 4369 (time 1782756656.24 total 70.10 last 61.93)
 - open/close 4935 (time 1782756666.25 total 80.11 last 56.55)
 - open/close 5534 (time 1782756676.26 total 90.11 last 59.86)
 - open/close 6189 (time 1782756686.28 total 100.13 last 65.39)
 - open/close 6757 (time 1782756696.28 total 110.14 last 56.76)
 - open/close 7315 (time 1782756706.29 total 120.14 last 55.76)
 - open/close 7878 (time 1782756716.29 total 130.15 last 56.30)
 - open/close 8475 (time 1782756726.29 total 140.15 last 59.68)
 - open/close 9135 (time 1782756736.30 total 150.16 last 65.94)
 - open/close 9770 (time 1782756746.31 total 160.17 last 63.44)
total: 10000 open/close in 163.80 seconds: 61.05 ops/second
 - unlinked 0 (time 1782756754 ; total 0 ; last 0)
total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second
 - open/close 574 (time 1782756891.23 total 10.01 last 57.35)
 - open/close 1105 (time 1782756901.23 total 20.02 last 53.05)
 - open/close 1765 (time 1782756911.24 total 30.02 last 65.95)
 - open/close 2351 (time 1782756921.24 total 40.03 last 58.58)
 - open/close 2877 (time 1782756931.24 total 50.03 last 52.60)
 - open/close 3353 (time 1782756941.25 total 60.03 last 47.57)
 - open/close 3980 (time 1782756951.27 total 70.06 last 62.56)
 - open/close 4578 (time 1782756961.28 total 80.06 last 59.75)
 - open/close 5210 (time 1782756971.29 total 90.07 last 63.17)
 - open/close 5889 (time 1782756981.29 total 100.07 last 67.90)
 - open/close 6549 (time 1782756991.29 total 110.07 last 65.99)
 - open/close 7169 (time 1782757001.29 total 120.07 last 61.99)
 - open/close 7818 (time 1782757011.32 total 130.10 last 64.70)
 - open/close 8465 (time 1782757021.33 total 140.12 last 64.62)
 - open/close 9087 (time 1782757031.33 total 150.12 last 62.19)
 - open/close 9671 (time 1782757041.34 total 160.12 last 58.36)
total: 10000 open/close in 165.67 seconds: 60.36 ops/second
 - unlinked 0 (time 1782757051 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 453 (time 1782757176.89 total 10.01 last 45.27)
 - open/close 988 (time 1782757186.91 total 20.02 last 53.41)
 - open/close 1605 (time 1782757196.92 total 30.04 last 61.62)
 - open/close 2202 (time 1782757206.93 total 40.04 last 59.67)
 - open/close 2754 (time 1782757216.94 total 50.06 last 55.12)
 - open/close 3157 (time 1782757226.95 total 60.06 last 40.28)
 - open/close 3556 (time 1782757236.96 total 70.08 last 39.85)
 - open/close 4048 (time 1782757246.97 total 80.09 last 49.15)
 - open/close 4652 (time 1782757256.98 total 90.09 last 60.37)
 - open/close 5285 (time 1782757266.98 total 100.09 last 63.29)
 - open/close 5870 (time 1782757276.99 total 110.11 last 58.41)
 - open/close 6363 (time 1782757287.00 total 120.12 last 49.25)
 - open/close 6853 (time 1782757297.01 total 130.12 last 48.99)
 - open/close 7383 (time 1782757307.03 total 140.14 last 52.88)
 - open/close 8001 (time 1782757317.03 total 150.15 last 61.76)
 - open/close 8646 (time 1782757327.04 total 160.15 last 64.47)
 - open/close 9191 (time 1782757337.07 total 170.18 last 54.34)
 - open/close 9808 (time 1782757347.07 total 180.19 last 61.68)
total: 10000 open/close in 183.44 seconds: 54.51 ops/second
 - unlinked 0 (time 1782757354 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 553 (time 1782757473.55 total 10.01 last 55.27)
 - open/close 1140 (time 1782757483.55 total 20.01 last 58.70)
 - open/close 1764 (time 1782757493.55 total 30.01 last 62.38)
 - open/close 2408 (time 1782757503.55 total 40.01 last 64.39)
 - open/close 3001 (time 1782757513.55 total 50.01 last 59.29)
 - open/close 3632 (time 1782757523.56 total 60.01 last 63.08)
 - open/close 4290 (time 1782757533.57 total 70.02 last 65.74)
 - open/close 4885 (time 1782757543.58 total 80.03 last 59.43)
 - open/close 5535 (time 1782757553.59 total 90.05 last 64.92)
 - open/close 6144 (time 1782757563.60 total 100.06 last 60.80)
 - open/close 6756 (time 1782757573.61 total 110.07 last 61.17)
 - open/close 7321 (time 1782757583.61 total 120.07 last 56.49)
 - open/close 7901 (time 1782757593.62 total 130.08 last 57.93)
 - open/close 8330 (time 1782757603.65 total 140.11 last 42.79)
 - open/close 8814 (time 1782757613.66 total 150.12 last 48.35)
 - open/close 9273 (time 1782757623.66 total 160.12 last 45.89)
 - open/close 9729 (time 1782757633.67 total 170.13 last 45.57)
total: 10000 open/close in 175.60 seconds: 56.95 ops/second
 - unlinked 0 (time 1782757643 ; total 0 ; last 0)
total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second
 - open/close 510 (time 1782757775.85 total 10.01 last 50.94)
 - open/close 1096 (time 1782757785.87 total 20.03 last 58.47)
 - open/close 1698 (time 1782757795.88 total 30.04 last 60.18)
 - open/close 2299 (time 1782757805.89 total 40.05 last 60.04)
 - open/close 2864 (time 1782757815.89 total 50.05 last 56.49)
 - open/close 3437 (time 1782757825.89 total 60.05 last 57.28)
 - open/close 4083 (time 1782757835.91 total 70.07 last 64.50)
 - open/close 4693 (time 1782757845.92 total 80.08 last 60.94)
 - open/close 5306 (time 1782757855.93 total 90.09 last 61.21)
 - open/close 5934 (time 1782757865.95 total 100.11 last 62.71)
 - open/close 6471 (time 1782757875.95 total 110.11 last 53.69)
 - open/close 7011 (time 1782757885.95 total 120.11 last 53.99)
 - open/close 7491 (time 1782757895.95 total 130.12 last 47.97)
 - open/close 8075 (time 1782757905.96 total 140.12 last 58.36)
 - open/close 8628 (time 1782757915.98 total 150.14 last 55.21)
 - open/close 9142 (time 1782757925.98 total 160.14 last 51.37)
 - open/close 9796 (time 1782757935.99 total 170.16 last 65.33)
total: 10000 open/close in 173.59 seconds: 57.61 ops/second
 - unlinked 0 (time 1782757944 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 406 (time 1782758071.94 total 10.01 last 40.56)
 - open/close 816 (time 1782758081.95 total 20.02 last 40.98)
 - open/close 1269 (time 1782758091.96 total 30.03 last 45.24)
 - open/close 1833 (time 1782758101.97 total 40.03 last 56.39)
 - open/close 2391 (time 1782758112.00 total 50.06 last 55.64)
 - open/close 2902 (time 1782758122.00 total 60.06 last 51.10)
 - open/close 3432 (time 1782758132.04 total 70.11 last 52.76)
 - open/close 3877 (time 1782758142.06 total 80.13 last 44.41)
 - open/close 4343 (time 1782758152.06 total 90.13 last 46.59)
 - open/close 4909 (time 1782758162.08 total 100.14 last 56.52)
 - open/close 5410 (time 1782758172.10 total 110.16 last 49.99)
 - open/close 5897 (time 1782758182.11 total 120.18 last 48.63)
 - open/close 6276 (time 1782758192.12 total 130.19 last 37.86)
 - open/close 6638 (time 1782758202.19 total 140.26 last 35.95)
 - open/close 7073 (time 1782758212.22 total 150.28 last 43.39)
 - open/close 7634 (time 1782758222.23 total 160.29 last 56.03)
 - open/close 8248 (time 1782758232.24 total 170.30 last 61.33)
 - open/close 8840 (time 1782758242.25 total 180.32 last 59.12)
 - open/close 9441 (time 1782758252.26 total 190.32 last 60.07)
 - open/close 9926 (time 1782758262.28 total 200.34 last 48.41)
total: 10000 open/close in 202.07 seconds: 49.49 ops/second
 - unlinked 0 (time 1782758272 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 538 (time 1782758394.55 total 10.01 last 53.75)
 - open/close 1097 (time 1782758404.56 total 20.02 last 55.86)
 - open/close 1656 (time 1782758414.57 total 30.02 last 55.86)
 - open/close 2219 (time 1782758424.58 total 40.03 last 56.24)
 - open/close 2798 (time 1782758434.58 total 50.04 last 57.89)
 - open/close 3365 (time 1782758444.59 total 60.05 last 56.63)
 - open/close 3919 (time 1782758454.62 total 70.08 last 55.24)
 - open/close 4463 (time 1782758464.62 total 80.08 last 54.39)
 - open/close 4992 (time 1782758474.63 total 90.08 last 52.87)
 - open/close 5652 (time 1782758484.63 total 100.09 last 65.99)
 - open/close 6251 (time 1782758494.64 total 110.10 last 59.84)
 - open/close 6863 (time 1782758504.64 total 120.10 last 61.18)
 - open/close 7523 (time 1782758514.64 total 130.10 last 66.00)
 - open/close 8196 (time 1782758524.65 total 140.11 last 67.22)
 - open/close 8839 (time 1782758534.67 total 150.13 last 64.19)
 - open/close 9425 (time 1782758544.69 total 160.15 last 58.47)
total: 10000 open/close in 169.25 seconds: 59.08 ops/second
 - unlinked 0 (time 1782758559 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 625 (time 1782758685.01 total 10.00 last 62.47)
 - open/close 1210 (time 1782758695.02 total 20.02 last 58.40)
 - open/close 1792 (time 1782758705.03 total 30.03 last 58.15)
 - open/close 2449 (time 1782758715.04 total 40.04 last 65.64)
 - open/close 3029 (time 1782758725.05 total 50.05 last 57.94)
 - open/close 3543 (time 1782758735.07 total 60.07 last 51.30)
 - open/close 3901 (time 1782758745.08 total 70.08 last 35.75)
 - open/close 4239 (time 1782758755.10 total 80.10 last 33.74)
 - open/close 4732 (time 1782758765.11 total 90.10 last 49.27)
 - open/close 5282 (time 1782758775.11 total 100.11 last 54.98)
 - open/close 5842 (time 1782758785.12 total 110.11 last 55.96)
 - open/close 6393 (time 1782758795.13 total 120.13 last 55.00)
 - open/close 6937 (time 1782758805.14 total 130.14 last 54.38)
 - open/close 7479 (time 1782758815.15 total 140.15 last 54.12)
 - open/close 8043 (time 1782758825.16 total 150.16 last 56.36)
 - open/close 8634 (time 1782758835.16 total 160.16 last 59.08)
 - open/close 9175 (time 1782758845.17 total 170.17 last 54.04)
 - open/close 9695 (time 1782758855.19 total 180.18 last 51.94)
total: 10000 open/close in 185.38 seconds: 53.94 ops/second
 - unlinked 0 (time 1782758866 ; total 0 ; last 0)
total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second
 - open/close 566 (time 1782758997.47 total 10.01 last 56.55)
 - open/close 1167 (time 1782759007.47 total 20.02 last 60.06)
 - open/close 1805 (time 1782759017.48 total 30.02 last 63.79)
 - open/close 2432 (time 1782759027.48 total 40.02 last 62.69)
 - open/close 2915 (time 1782759037.49 total 50.03 last 48.24)
 - open/close 3540 (time 1782759047.51 total 60.05 last 62.39)
 - open/close 4215 (time 1782759057.52 total 70.06 last 67.44)
 - open/close 4876 (time 1782759067.54 total 80.08 last 65.93)
 - open/close 5625 (time 1782759077.55 total 90.09 last 74.83)
 - open/close 6397 (time 1782759087.55 total 100.09 last 77.19)
 - open/close 7145 (time 1782759097.56 total 110.10 last 74.79)
 - open/close 7882 (time 1782759107.56 total 120.10 last 73.67)
 - open/close 8624 (time 1782759117.57 total 130.11 last 74.16)
 - open/close 9377 (time 1782759127.58 total 140.12 last 75.18)
total: 10000 open/close in 148.24 seconds: 67.46 ops/second
 - unlinked 0 (time 1782759138 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 1300 (time 1782759219.55 total 10.00 last 129.95)
 - open/close 2703 (time 1782759229.55 total 20.01 last 140.27)
 - open/close 3859 (time 1782759239.55 total 30.01 last 115.58)
 - open/close 4974 (time 1782759249.56 total 40.01 last 111.45)
 - open/close 6207 (time 1782759259.56 total 50.02 last 123.23)
 - open/close 7411 (time 1782759269.57 total 60.02 last 120.36)
 - open/close 8846 (time 1782759279.57 total 70.03 last 143.43)
total: 10000 open/close in 79.79 seconds: 125.34 ops/second
 - unlinked 0 (time 1782759291 ; total 0 ; last 0)
total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg417-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg417-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.117@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg417-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Started lustre-OST0000
oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg417-client.virtnet:  -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre
On OST0, 19414 used inodes rc=0
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg417-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg417-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.117@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.117@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.117@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.117@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg417-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Started lustre-OST0000
oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg417-client.virtnet:  -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg417-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg417-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server
stop mds service on oleg417-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server

unloading modules via unload_modules_local on: 'oleg417-server'
oleg417-server: oleg417-server.virtnet: executing unload_modules_local
oleg417-server: 
modules unloaded.
pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2
pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2