== conf-sanity test 69: replace an OST with the same index ========================================================== 15:40:24 (1775763624)
start mds service on oleg459-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing load_modules_local
oleg459-server: Loading modules from /home/green/git/lustre-release/lustre
oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg459-server: detected 4 online CPUs by sysfs
oleg459-server: MODOPTS_LIBCFS=
oleg459-server: Force libcfs to create 2 CPU partitions
oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg459-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 544 (time 1775763706.82 total 10.01 last 54.32)
 - open/close 987 (time 1775763716.83 total 20.03 last 44.24)
 - open/close 1426 (time 1775763726.85 total 30.05 last 43.81)
 - open/close 1971 (time 1775763736.86 total 40.06 last 54.43)
 - open/close 2520 (time 1775763746.87 total 50.06 last 54.88)
 - open/close 3063 (time 1775763756.87 total 60.06 last 54.29)
 - open/close 3663 (time 1775763766.88 total 70.08 last 59.91)
 - open/close 4241 (time 1775763776.89 total 80.08 last 57.79)
 - open/close 4871 (time 1775763786.90 total 90.09 last 62.93)
 - open/close 5453 (time 1775763796.90 total 100.09 last 58.20)
 - open/close 6011 (time 1775763806.91 total 110.11 last 55.72)
 - open/close 6597 (time 1775763816.94 total 120.13 last 58.45)
 - open/close 7157 (time 1775763826.96 total 130.15 last 55.87)
 - open/close 7708 (time 1775763836.99 total 140.18 last 54.96)
 - open/close 8220 (time 1775763846.99 total 150.18 last 51.19)
 - open/close 8746 (time 1775763857.01 total 160.20 last 52.48)
 - open/close 9304 (time 1775763867.01 total 170.21 last 55.78)
 - open/close 9860 (time 1775763877.02 total 180.21 last 55.58)
total: 10000 open/close in 182.57 seconds: 54.77 ops/second
 - unlinked 0 (time 1775763883 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 604 (time 1775764002.76 total 10.02 last 60.30)
 - open/close 1243 (time 1775764012.77 total 20.02 last 63.86)
 - open/close 1865 (time 1775764022.77 total 30.02 last 62.19)
 - open/close 2457 (time 1775764032.77 total 40.03 last 59.18)
 - open/close 3007 (time 1775764042.78 total 50.03 last 54.97)
 - open/close 3583 (time 1775764052.79 total 60.04 last 57.56)
 - open/close 4169 (time 1775764062.79 total 70.05 last 58.57)
 - open/close 4763 (time 1775764072.79 total 80.05 last 59.39)
 - open/close 5358 (time 1775764082.82 total 90.07 last 59.36)
 - open/close 5944 (time 1775764092.82 total 100.08 last 58.56)
 - open/close 6490 (time 1775764102.83 total 110.09 last 54.56)
 - open/close 7100 (time 1775764112.83 total 120.09 last 61.00)
 - open/close 7603 (time 1775764122.83 total 130.09 last 50.30)
 - open/close 8126 (time 1775764132.85 total 140.10 last 52.22)
 - open/close 8756 (time 1775764142.87 total 150.13 last 62.84)
 - open/close 9326 (time 1775764152.88 total 160.14 last 56.94)
 - open/close 9901 (time 1775764162.89 total 170.14 last 57.48)
total: 10000 open/close in 171.84 seconds: 58.19 ops/second
 - unlinked 0 (time 1775764169 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 542 (time 1775764291.48 total 10.00 last 54.20)
 - open/close 1104 (time 1775764301.49 total 20.01 last 56.12)
 - open/close 1697 (time 1775764311.50 total 30.02 last 59.26)
 - open/close 2272 (time 1775764321.51 total 40.04 last 57.40)
 - open/close 2821 (time 1775764331.51 total 50.04 last 54.90)
 - open/close 3375 (time 1775764341.52 total 60.05 last 55.34)
 - open/close 3913 (time 1775764351.54 total 70.06 last 53.73)
 - open/close 4505 (time 1775764361.54 total 80.07 last 59.17)
 - open/close 5065 (time 1775764371.55 total 90.08 last 55.93)
 - open/close 5613 (time 1775764381.57 total 100.09 last 54.74)
 - open/close 6117 (time 1775764391.57 total 110.10 last 50.37)
 - open/close 6630 (time 1775764401.61 total 120.14 last 51.10)
 - open/close 7235 (time 1775764411.61 total 130.14 last 60.49)
 - open/close 7800 (time 1775764421.62 total 140.14 last 56.47)
 - open/close 8364 (time 1775764431.64 total 150.16 last 56.29)
 - open/close 8926 (time 1775764441.65 total 160.18 last 56.12)
 - open/close 9441 (time 1775764451.67 total 170.19 last 51.42)
 - open/close 9925 (time 1775764461.67 total 180.20 last 48.38)
total: 10000 open/close in 181.65 seconds: 55.05 ops/second
 - unlinked 0 (time 1775764467 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
 - open/close 585 (time 1775764595.86 total 10.01 last 58.46)
 - open/close 1167 (time 1775764605.88 total 20.03 last 58.05)
 - open/close 1681 (time 1775764615.90 total 30.05 last 51.30)
 - open/close 2269 (time 1775764625.92 total 40.07 last 58.69)
 - open/close 2865 (time 1775764635.93 total 50.08 last 59.54)
 - open/close 3426 (time 1775764645.94 total 60.09 last 56.05)
 - open/close 3971 (time 1775764655.95 total 70.10 last 54.47)
 - open/close 4522 (time 1775764665.96 total 80.11 last 55.05)
 - open/close 5136 (time 1775764675.96 total 90.11 last 61.36)
 - open/close 5650 (time 1775764685.98 total 100.13 last 51.32)
 - open/close 6131 (time 1775764696.03 total 110.18 last 47.86)
 - open/close 6616 (time 1775764706.04 total 120.19 last 48.45)
 - open/close 7167 (time 1775764716.05 total 130.19 last 55.06)
 - open/close 7740 (time 1775764726.06 total 140.21 last 57.22)
 - open/close 8278 (time 1775764736.08 total 150.23 last 53.70)
 - open/close 8888 (time 1775764746.08 total 160.23 last 60.97)
 - open/close 9455 (time 1775764756.10 total 170.25 last 56.60)
total: 10000 open/close in 179.89 seconds: 55.59 ops/second
 - unlinked 0 (time 1775764770 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 547 (time 1775764894.18 total 10.03 last 54.53)
 - open/close 1100 (time 1775764904.18 total 20.03 last 55.29)
 - open/close 1692 (time 1775764914.19 total 30.04 last 59.19)
 - open/close 2298 (time 1775764924.19 total 40.04 last 60.57)
 - open/close 2829 (time 1775764934.20 total 50.05 last 53.05)
 - open/close 3329 (time 1775764944.22 total 60.07 last 49.89)
 - open/close 3924 (time 1775764954.23 total 70.07 last 59.48)
 - open/close 4481 (time 1775764964.23 total 80.07 last 55.70)
 - open/close 5019 (time 1775764974.23 total 90.08 last 53.79)
 - open/close 5574 (time 1775764984.23 total 100.08 last 55.50)
 - open/close 6150 (time 1775764994.24 total 110.09 last 57.55)
 - open/close 6728 (time 1775765004.26 total 120.10 last 57.70)
 - open/close 7249 (time 1775765014.28 total 130.13 last 51.98)
 - open/close 7823 (time 1775765024.28 total 140.13 last 57.39)
 - open/close 8384 (time 1775765034.28 total 150.13 last 56.09)
 - open/close 8969 (time 1775765044.30 total 160.15 last 58.41)
 - open/close 9498 (time 1775765054.32 total 170.17 last 52.79)
total: 10000 open/close in 178.88 seconds: 55.90 ops/second
 - unlinked 0 (time 1775765067 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 600 (time 1775765189.09 total 10.01 last 59.93)
 - open/close 1176 (time 1775765199.10 total 20.03 last 57.52)
 - open/close 1802 (time 1775765209.11 total 30.04 last 62.52)
 - open/close 2396 (time 1775765219.13 total 40.06 last 59.30)
 - open/close 2959 (time 1775765229.13 total 50.06 last 56.29)
 - open/close 3501 (time 1775765239.14 total 60.06 last 54.16)
 - open/close 4066 (time 1775765249.15 total 70.08 last 56.43)
 - open/close 4638 (time 1775765259.16 total 80.08 last 57.15)
 - open/close 5201 (time 1775765269.17 total 90.09 last 56.26)
 - open/close 5779 (time 1775765279.17 total 100.10 last 57.76)
 - open/close 6346 (time 1775765289.18 total 110.11 last 56.65)
 - open/close 6939 (time 1775765299.18 total 120.11 last 59.29)
 - open/close 7536 (time 1775765309.20 total 130.13 last 59.60)
 - open/close 8114 (time 1775765319.24 total 140.16 last 57.60)
 - open/close 8716 (time 1775765329.24 total 150.16 last 60.18)
 - open/close 9285 (time 1775765339.24 total 160.16 last 56.90)
 - open/close 9908 (time 1775765349.24 total 170.17 last 62.28)
total: 10000 open/close in 171.69 seconds: 58.25 ops/second
 - unlinked 0 (time 1775765355 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 593 (time 1775765471.84 total 10.00 last 59.27)
 - open/close 1181 (time 1775765481.86 total 20.02 last 58.70)
 - open/close 1788 (time 1775765491.88 total 30.05 last 60.56)
 - open/close 2433 (time 1775765501.89 total 40.05 last 64.46)
 - open/close 3105 (time 1775765511.89 total 50.06 last 67.16)
 - open/close 3747 (time 1775765521.91 total 60.07 last 64.11)
 - open/close 4370 (time 1775765531.92 total 70.08 last 62.23)
 - open/close 4921 (time 1775765541.93 total 80.09 last 55.06)
 - open/close 5551 (time 1775765551.94 total 90.10 last 62.92)
 - open/close 6147 (time 1775765561.95 total 100.11 last 59.54)
 - open/close 6734 (time 1775765571.96 total 110.12 last 58.64)
 - open/close 7328 (time 1775765581.96 total 120.12 last 59.38)
 - open/close 7919 (time 1775765591.97 total 130.13 last 59.07)
 - open/close 8521 (time 1775765601.99 total 140.16 last 60.04)
 - open/close 8958 (time 1775765612.02 total 150.18 last 43.60)
 - open/close 9462 (time 1775765622.02 total 160.19 last 50.36)
 - open/close 9993 (time 1775765632.02 total 170.19 last 53.10)
total: 10000 open/close in 170.27 seconds: 58.73 ops/second
 - unlinked 0 (time 1775765636 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 617 (time 1775765753.02 total 10.01 last 61.62)
 - open/close 1240 (time 1775765763.03 total 20.02 last 62.24)
 - open/close 1820 (time 1775765773.04 total 30.04 last 57.93)
 - open/close 2408 (time 1775765783.05 total 40.04 last 58.76)
 - open/close 2970 (time 1775765793.06 total 50.06 last 56.13)
 - open/close 3536 (time 1775765803.08 total 60.07 last 56.51)
 - open/close 4158 (time 1775765813.09 total 70.08 last 62.13)
 - open/close 4711 (time 1775765823.09 total 80.09 last 55.28)
 - open/close 5266 (time 1775765833.10 total 90.09 last 55.46)
 - open/close 5816 (time 1775765843.11 total 100.10 last 54.94)
 - open/close 6339 (time 1775765853.11 total 110.11 last 52.28)
 - open/close 6981 (time 1775765863.12 total 120.12 last 64.14)
 - open/close 7688 (time 1775765873.13 total 130.13 last 70.64)
 - open/close 8276 (time 1775765883.14 total 140.13 last 58.77)
 - open/close 8900 (time 1775765893.15 total 150.14 last 62.37)
 - open/close 9467 (time 1775765903.15 total 160.15 last 56.64)
 - open/close 9961 (time 1775765913.16 total 170.16 last 49.37)
total: 10000 open/close in 171.00 seconds: 58.48 ops/second
 - unlinked 0 (time 1775765920 ; total 0 ; last 0)
total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second
 - open/close 426 (time 1775766060.91 total 10.01 last 42.57)
 - open/close 934 (time 1775766070.91 total 20.01 last 50.80)
 - open/close 1425 (time 1775766080.92 total 30.02 last 49.05)
 - open/close 1871 (time 1775766090.95 total 40.04 last 44.48)
 - open/close 2368 (time 1775766100.96 total 50.05 last 49.65)
 - open/close 2897 (time 1775766110.97 total 60.07 last 52.81)
 - open/close 3345 (time 1775766120.98 total 70.08 last 44.76)
 - open/close 3717 (time 1775766131.01 total 80.11 last 37.10)
 - open/close 4135 (time 1775766141.02 total 90.11 last 41.78)
 - open/close 4623 (time 1775766151.06 total 100.15 last 48.61)
 - open/close 5108 (time 1775766161.06 total 110.15 last 48.49)
 - open/close 5628 (time 1775766171.06 total 120.16 last 51.98)
 - open/close 6106 (time 1775766181.07 total 130.17 last 47.74)
 - open/close 6549 (time 1775766191.09 total 140.19 last 44.21)
 - open/close 6909 (time 1775766201.12 total 150.21 last 35.91)
 - open/close 7339 (time 1775766211.14 total 160.24 last 42.90)
 - open/close 7765 (time 1775766221.15 total 170.24 last 42.57)
 - open/close 8281 (time 1775766231.16 total 180.25 last 51.55)
 - open/close 8812 (time 1775766241.17 total 190.27 last 53.03)
 - open/close 9380 (time 1775766251.18 total 200.27 last 56.75)
 - open/close 9915 (time 1775766261.18 total 210.28 last 53.50)
total: 10000 open/close in 211.96 seconds: 47.18 ops/second
 - unlinked 0 (time 1775766269 ; total 0 ; last 0)
total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second
 - open/close 552 (time 1775766398.75 total 10.01 last 55.15)
 - open/close 1062 (time 1775766408.75 total 20.02 last 50.96)
 - open/close 1663 (time 1775766418.77 total 30.03 last 60.02)
 - open/close 2278 (time 1775766428.78 total 40.05 last 61.40)
 - open/close 2836 (time 1775766438.79 total 50.06 last 55.74)
 - open/close 3377 (time 1775766448.82 total 60.08 last 53.97)
 - open/close 3925 (time 1775766458.83 total 70.09 last 54.75)
 - open/close 4435 (time 1775766468.83 total 80.09 last 50.99)
 - open/close 4970 (time 1775766478.84 total 90.11 last 53.42)
 - open/close 5531 (time 1775766488.85 total 100.12 last 56.05)
 - open/close 6060 (time 1775766498.86 total 110.12 last 52.85)
 - open/close 6617 (time 1775766508.87 total 120.13 last 55.65)
 - open/close 7156 (time 1775766518.89 total 130.15 last 53.81)
 - open/close 7704 (time 1775766528.89 total 140.16 last 54.76)
 - open/close 8271 (time 1775766538.90 total 150.16 last 56.67)
 - open/close 8803 (time 1775766548.90 total 160.17 last 53.18)
 - open/close 9320 (time 1775766558.90 total 170.17 last 51.69)
 - open/close 9869 (time 1775766568.90 total 180.17 last 54.90)
total: 10000 open/close in 182.49 seconds: 54.80 ops/second
 - unlinked 0 (time 1775766576 ; total 0 ; last 0)
total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.159@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.159@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg459-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-OST0000
oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg459-client.virtnet:  -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg459-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg459-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server
stop mds service on oleg459-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server

unloading modules via unload_modules_local on: 'oleg459-server'
oleg459-server: oleg459-server.virtnet: executing unload_modules_local
oleg459-server: 
modules unloaded.
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2