== conf-sanity test 69: replace an OST with the same index ========================================================== 19:41:58 (1785886918)
start mds service on oleg203-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg203-server'
oleg203-server: oleg203-server.virtnet: executing load_modules_local
oleg203-server: Loading modules from /home/green/git/lustre-release/lustre
oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg203-server: detected 4 online CPUs by sysfs
oleg203-server: MODOPTS_LIBCFS=
oleg203-server: Force libcfs to create 2 CPU partitions
oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 533 (time 1785887040.59 total 10.02 last 53.18)
 - open/close 1112 (time 1785887050.59 total 20.03 last 57.87)
 - open/close 1514 (time 1785887060.62 total 30.05 last 40.10)
 - open/close 2258 (time 1785887070.64 total 40.08 last 74.23)
 - open/close 2669 (time 1785887080.67 total 50.11 last 40.97)
 - open/close 3201 (time 1785887090.69 total 60.13 last 53.08)
 - open/close 3667 (time 1785887100.73 total 70.16 last 46.45)
 - open/close 4063 (time 1785887110.75 total 80.19 last 39.49)
 - open/close 4663 (time 1785887120.76 total 90.19 last 59.99)
 - open/close 5368 (time 1785887130.76 total 100.20 last 70.47)
 - open/close 5944 (time 1785887140.78 total 110.21 last 57.49)
 - open/close 6459 (time 1785887150.79 total 120.22 last 51.45)
 - open/close 6767 (time 1785887161.42 total 130.86 last 28.97)
 - open/close 7037 (time 1785887171.42 total 140.86 last 26.99)
 - open/close 7154 (time 1785887181.43 total 150.87 last 11.69)
 - open/close 7398 (time 1785887191.45 total 160.88 last 24.37)
 - open/close 7848 (time 1785887201.46 total 170.90 last 44.93)
 - open/close 8540 (time 1785887211.47 total 180.91 last 69.12)
 - open/close 9192 (time 1785887221.50 total 190.94 last 65.03)
 - open/close 9792 (time 1785887231.50 total 200.94 last 59.98)
total: 10000 open/close in 203.25 seconds: 49.20 ops/second
 - unlinked 0 (time 1785887239 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 951 (time 1785887362.36 total 10.02 last 94.95)
 - open/close 1545 (time 1785887372.38 total 20.04 last 59.27)
 - open/close 2281 (time 1785887382.38 total 30.04 last 73.55)
 - open/close 2906 (time 1785887392.39 total 40.05 last 62.46)
 - open/close 3470 (time 1785887402.39 total 50.05 last 56.39)
 - open/close 4152 (time 1785887412.40 total 60.06 last 68.19)
 - open/close 4788 (time 1785887422.41 total 70.07 last 63.49)
 - open/close 5446 (time 1785887432.42 total 80.08 last 65.78)
 - open/close 6196 (time 1785887442.42 total 90.08 last 74.96)
 - open/close 6939 (time 1785887452.42 total 100.08 last 74.30)
 - open/close 7375 (time 1785887462.43 total 110.08 last 43.59)
 - open/close 8181 (time 1785887472.43 total 120.09 last 80.60)
 - open/close 8976 (time 1785887482.43 total 130.09 last 79.44)
 - open/close 9512 (time 1785887492.46 total 140.12 last 53.45)
total: 10000 open/close in 147.41 seconds: 67.84 ops/second
 - unlinked 0 (time 1785887504 ; total 0 ; last 0)
total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second
 - open/close 520 (time 1785887644.81 total 10.01 last 51.97)
 - open/close 1150 (time 1785887654.81 total 20.01 last 62.97)
 - open/close 1938 (time 1785887664.82 total 30.01 last 78.77)
 - open/close 2585 (time 1785887674.82 total 40.02 last 64.65)
 - open/close 3089 (time 1785887684.84 total 50.03 last 50.33)
 - open/close 3566 (time 1785887694.95 total 60.15 last 47.14)
 - open/close 3978 (time 1785887704.96 total 70.16 last 41.19)
 - open/close 4417 (time 1785887714.99 total 80.18 last 43.77)
 - open/close 4920 (time 1785887725.01 total 90.21 last 50.16)
 - open/close 5483 (time 1785887735.02 total 100.22 last 56.26)
 - open/close 5880 (time 1785887745.03 total 110.23 last 39.66)
 - open/close 6353 (time 1785887755.04 total 120.24 last 47.27)
 - open/close 6963 (time 1785887765.04 total 130.24 last 60.99)
 - open/close 7741 (time 1785887775.07 total 140.27 last 77.57)
 - open/close 8358 (time 1785887785.08 total 150.27 last 61.66)
 - open/close 8777 (time 1785887795.08 total 160.28 last 41.87)
 - open/close 9334 (time 1785887805.10 total 170.30 last 55.62)
total: 10000 open/close in 180.11 seconds: 55.52 ops/second
 - unlinked 0 (time 1785887820 ; total 0 ; last 0)
total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second
 - open/close 479 (time 1785887951.77 total 10.01 last 47.87)
 - open/close 931 (time 1785887961.78 total 20.01 last 45.20)
 - open/close 1753 (time 1785887971.78 total 30.01 last 82.14)
 - open/close 2370 (time 1785887981.78 total 40.02 last 61.69)
 - open/close 2897 (time 1785887991.85 total 50.09 last 52.34)
 - open/close 3410 (time 1785888001.87 total 60.11 last 51.20)
 - open/close 3791 (time 1785888011.93 total 70.16 last 37.90)
 - open/close 4133 (time 1785888021.97 total 80.20 last 34.05)
 - open/close 4642 (time 1785888032.00 total 90.23 last 50.76)
 - open/close 5397 (time 1785888042.00 total 100.23 last 75.48)
 - open/close 5984 (time 1785888052.00 total 110.23 last 58.70)
 - open/close 6561 (time 1785888062.01 total 120.24 last 57.67)
 - open/close 6952 (time 1785888072.02 total 130.25 last 39.05)
 - open/close 7418 (time 1785888082.02 total 140.25 last 46.59)
 - open/close 8073 (time 1785888092.03 total 150.27 last 65.41)
 - open/close 8796 (time 1785888102.04 total 160.27 last 72.27)
 - open/close 9463 (time 1785888112.07 total 170.30 last 66.48)
 - open/close 9970 (time 1785888122.07 total 180.31 last 50.69)
total: 10000 open/close in 180.86 seconds: 55.29 ops/second
 - unlinked 0 (time 1785888128 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 568 (time 1785888255.07 total 10.00 last 56.78)
 - open/close 1006 (time 1785888265.07 total 20.01 last 43.79)
 - open/close 1442 (time 1785888275.08 total 30.01 last 43.57)
 - open/close 1991 (time 1785888285.11 total 40.05 last 54.73)
 - open/close 2949 (time 1785888295.11 total 50.05 last 95.77)
 - open/close 3627 (time 1785888305.12 total 60.06 last 67.76)
 - open/close 4361 (time 1785888315.14 total 70.07 last 73.27)
 - open/close 4856 (time 1785888325.15 total 80.08 last 49.46)
 - open/close 5253 (time 1785888335.18 total 90.12 last 39.56)
 - open/close 5801 (time 1785888345.19 total 100.12 last 54.77)
 - open/close 6524 (time 1785888355.20 total 110.13 last 72.21)
 - open/close 7379 (time 1785888365.20 total 120.14 last 85.46)
 - open/close 8015 (time 1785888375.23 total 130.16 last 63.44)
 - open/close 8513 (time 1785888385.26 total 140.19 last 49.67)
 - open/close 8983 (time 1785888395.26 total 150.20 last 46.96)
 - open/close 9447 (time 1785888405.28 total 160.21 last 46.33)
total: 10000 open/close in 170.11 seconds: 58.78 ops/second
 - unlinked 0 (time 1785888422 ; total 0 ; last 0)
total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second
 - open/close 639 (time 1785888553.31 total 10.01 last 63.80)
 - open/close 1242 (time 1785888563.31 total 20.02 last 60.29)
 - open/close 1736 (time 1785888573.34 total 30.05 last 49.24)
 - open/close 2082 (time 1785888583.34 total 40.05 last 34.59)
 - open/close 2550 (time 1785888593.36 total 50.07 last 46.72)
 - open/close 3072 (time 1785888603.36 total 60.07 last 52.17)
 - open/close 3743 (time 1785888613.38 total 70.09 last 67.01)
 - open/close 4216 (time 1785888623.40 total 80.11 last 47.22)
 - open/close 4763 (time 1785888633.41 total 90.12 last 54.60)
 - open/close 5128 (time 1785888643.44 total 100.15 last 36.42)
 - open/close 5705 (time 1785888653.45 total 110.16 last 57.65)
 - open/close 6343 (time 1785888663.45 total 120.16 last 63.75)
 - open/close 6743 (time 1785888673.47 total 130.18 last 39.94)
 - open/close 7244 (time 1785888683.47 total 140.18 last 50.10)
 - open/close 7674 (time 1785888693.49 total 150.20 last 42.93)
 - open/close 8355 (time 1785888703.50 total 160.21 last 68.01)
 - open/close 9058 (time 1785888713.51 total 170.22 last 70.21)
 - open/close 9969 (time 1785888723.51 total 180.22 last 91.10)
total: 10000 open/close in 180.50 seconds: 55.40 ops/second
 - unlinked 0 (time 1785888729 ; total 0 ; last 0)
total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second
 - open/close 444 (time 1785888856.73 total 10.01 last 44.34)
 - open/close 1033 (time 1785888866.75 total 20.03 last 58.81)
 - open/close 1763 (time 1785888876.75 total 30.03 last 72.99)
 - open/close 2452 (time 1785888886.75 total 40.04 last 68.85)
 - open/close 3126 (time 1785888896.76 total 50.04 last 67.39)
 - open/close 3841 (time 1785888906.80 total 60.08 last 71.18)
 - open/close 4447 (time 1785888916.81 total 70.09 last 60.57)
 - open/close 4970 (time 1785888926.82 total 80.11 last 52.21)
 - open/close 5448 (time 1785888936.85 total 90.13 last 47.70)
 - open/close 6261 (time 1785888946.85 total 100.13 last 81.29)
 - open/close 7096 (time 1785888956.86 total 110.14 last 83.41)
 - open/close 7808 (time 1785888966.86 total 120.15 last 71.15)
 - open/close 8483 (time 1785888976.89 total 130.17 last 67.35)
 - open/close 9021 (time 1785888986.89 total 140.18 last 53.75)
 - open/close 9676 (time 1785888996.90 total 150.18 last 65.46)
total: 10000 open/close in 156.00 seconds: 64.10 ops/second
 - unlinked 0 (time 1785889008 ; total 0 ; last 0)
total: 10000 unlinks in 121 seconds: 82.644630 unlinks/second
 - open/close 625 (time 1785889145.92 total 10.04 last 62.25)
 - open/close 1216 (time 1785889155.95 total 20.07 last 58.95)
 - open/close 2003 (time 1785889165.95 total 30.07 last 78.65)
 - open/close 2749 (time 1785889175.98 total 40.10 last 74.42)
 - open/close 3284 (time 1785889185.99 total 50.11 last 53.42)
 - open/close 3861 (time 1785889196.01 total 60.13 last 57.57)
 - open/close 4365 (time 1785889206.02 total 70.15 last 50.34)
 - open/close 5061 (time 1785889216.07 total 80.19 last 69.27)
 - open/close 5758 (time 1785889226.10 total 90.22 last 69.51)
 - open/close 6456 (time 1785889236.12 total 100.24 last 69.64)
 - open/close 7143 (time 1785889246.14 total 110.26 last 68.61)
 - open/close 7617 (time 1785889256.15 total 120.27 last 47.35)
 - open/close 8225 (time 1785889266.15 total 130.27 last 60.80)
 - open/close 9081 (time 1785889276.16 total 140.28 last 85.53)
 - open/close 9689 (time 1785889286.16 total 150.28 last 60.80)
total: 10000 open/close in 153.85 seconds: 65.00 ops/second
 - unlinked 0 (time 1785889295 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 529 (time 1785889419.17 total 10.02 last 52.79)
 - open/close 877 (time 1785889429.17 total 20.03 last 34.77)
 - open/close 1464 (time 1785889439.18 total 30.03 last 58.68)
 - open/close 1905 (time 1785889449.18 total 40.03 last 44.09)
 - open/close 2491 (time 1785889459.19 total 50.04 last 58.55)
 - open/close 2897 (time 1785889469.22 total 60.07 last 40.48)
 - open/close 3340 (time 1785889479.23 total 70.08 last 44.27)
 - open/close 4023 (time 1785889489.23 total 80.09 last 68.26)
 - open/close 4553 (time 1785889499.24 total 90.09 last 52.96)
 - open/close 5159 (time 1785889509.28 total 100.14 last 60.35)
 - open/close 5646 (time 1785889519.35 total 110.20 last 48.38)
 - open/close 6169 (time 1785889529.36 total 120.21 last 52.25)
 - open/close 6481 (time 1785889539.39 total 130.24 last 31.09)
 - open/close 7078 (time 1785889549.39 total 140.24 last 59.70)
 - open/close 7561 (time 1785889559.41 total 150.26 last 48.21)
 - open/close 8034 (time 1785889569.43 total 160.29 last 47.19)
 - open/close 8783 (time 1785889579.45 total 170.31 last 74.75)
 - open/close 9474 (time 1785889589.46 total 180.31 last 69.08)
 - open/close 9895 (time 1785889599.51 total 190.37 last 41.86)
total: 10000 open/close in 193.25 seconds: 51.75 ops/second
 - unlinked 0 (time 1785889610 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 410 (time 1785889727.18 total 10.00 last 40.99)
 - open/close 1008 (time 1785889737.18 total 20.00 last 59.79)
 - open/close 1631 (time 1785889747.18 total 30.01 last 62.28)
 - open/close 2143 (time 1785889757.19 total 40.01 last 51.17)
 - open/close 2627 (time 1785889767.21 total 50.03 last 48.30)
 - open/close 3070 (time 1785889777.21 total 60.04 last 44.28)
 - open/close 3431 (time 1785889787.22 total 70.05 last 36.06)
 - open/close 3854 (time 1785889797.23 total 80.06 last 42.26)
 - open/close 4570 (time 1785889807.24 total 90.06 last 71.55)
 - open/close 5090 (time 1785889817.24 total 100.06 last 52.00)
 - open/close 5751 (time 1785889827.24 total 110.07 last 66.08)
 - open/close 6377 (time 1785889837.25 total 120.07 last 62.58)
 - open/close 6871 (time 1785889847.26 total 130.08 last 49.35)
 - open/close 7457 (time 1785889857.26 total 140.08 last 58.59)
 - open/close 8065 (time 1785889867.27 total 150.10 last 60.72)
 - open/close 8963 (time 1785889877.28 total 160.11 last 89.68)
 - open/close 9507 (time 1785889887.30 total 170.13 last 54.31)
 - open/close 9969 (time 1785889897.33 total 180.15 last 46.08)
total: 10000 open/close in 181.09 seconds: 55.22 ops/second
 - unlinked 0 (time 1785889904 ; total 0 ; last 0)
total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.103@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.103@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.103@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.103@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.103@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg203-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-OST0000
oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg203-client.virtnet:  -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg203-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg203-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server
stop mds service on oleg203-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server

unloading modules via unload_modules_local on: 'oleg203-server'
oleg203-server: oleg203-server.virtnet: executing unload_modules_local
oleg203-server: 
modules unloaded.
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2