== conf-sanity test 69: replace an OST with the same index ========================================================== 02:40:37 (1785393637)
start mds service on oleg212-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg212-server'
oleg212-server: oleg212-server.virtnet: executing load_modules_local
oleg212-server: Loading modules from /home/green/git/lustre-release/lustre
oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg212-server: detected 4 online CPUs by sysfs
oleg212-server: MODOPTS_LIBCFS=
oleg212-server: Force libcfs to create 2 CPU partitions
oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg212-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg212-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1
Started lustre-OST0000
oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg212-client.virtnet:  -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 459 (time 1785393764.28 total 10.02 last 45.80)
 - open/close 1040 (time 1785393774.29 total 20.04 last 58.02)
 - open/close 1606 (time 1785393784.30 total 30.04 last 56.56)
 - open/close 2127 (time 1785393794.30 total 40.05 last 52.09)
 - open/close 2727 (time 1785393804.31 total 50.05 last 59.97)
 - open/close 3352 (time 1785393814.33 total 60.07 last 62.38)
 - open/close 3960 (time 1785393824.35 total 70.09 last 60.68)
 - open/close 4411 (time 1785393834.36 total 80.10 last 45.06)
 - open/close 4963 (time 1785393844.37 total 90.11 last 55.14)
 - open/close 5471 (time 1785393854.39 total 100.13 last 50.71)
 - open/close 6075 (time 1785393864.39 total 110.13 last 60.39)
 - open/close 6727 (time 1785393874.40 total 120.14 last 65.10)
 - open/close 7216 (time 1785393884.41 total 130.16 last 48.85)
 - open/close 7697 (time 1785393894.42 total 140.16 last 48.08)
 - open/close 8380 (time 1785393904.43 total 150.17 last 68.23)
 - open/close 8875 (time 1785393914.44 total 160.18 last 49.45)
 - open/close 9315 (time 1785393924.45 total 170.19 last 43.95)
 - open/close 9911 (time 1785393934.46 total 180.20 last 59.55)
total: 10000 open/close in 181.01 seconds: 55.25 ops/second
 - unlinked 0 (time 1785393939 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 540 (time 1785394050.35 total 10.01 last 53.93)
 - open/close 1089 (time 1785394060.35 total 20.02 last 54.87)
 - open/close 1504 (time 1785394070.38 total 30.05 last 41.38)
 - open/close 1909 (time 1785394080.40 total 40.06 last 40.43)
 - open/close 2440 (time 1785394090.41 total 50.07 last 53.07)
 - open/close 3002 (time 1785394100.41 total 60.07 last 56.17)
 - open/close 3450 (time 1785394110.42 total 70.08 last 44.77)
 - open/close 3870 (time 1785394120.45 total 80.11 last 41.87)
 - open/close 4364 (time 1785394130.46 total 90.12 last 49.34)
 - open/close 4946 (time 1785394140.47 total 100.13 last 58.16)
 - open/close 5516 (time 1785394150.48 total 110.14 last 56.95)
 - open/close 6010 (time 1785394160.48 total 120.14 last 49.40)
 - open/close 6527 (time 1785394170.49 total 130.15 last 51.65)
 - open/close 7169 (time 1785394180.49 total 140.16 last 64.17)
 - open/close 7757 (time 1785394190.51 total 150.17 last 58.70)
 - open/close 8258 (time 1785394200.53 total 160.19 last 50.01)
 - open/close 9029 (time 1785394210.56 total 170.23 last 76.83)
 - open/close 9797 (time 1785394220.57 total 180.24 last 76.72)
total: 10000 open/close in 183.02 seconds: 54.64 ops/second
 - unlinked 0 (time 1785394228 ; total 0 ; last 0)
total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second
 - open/close 423 (time 1785394364.48 total 10.01 last 42.27)
 - open/close 779 (time 1785394374.48 total 20.01 last 35.60)
 - open/close 1236 (time 1785394384.49 total 30.02 last 45.64)
 - open/close 1742 (time 1785394394.50 total 40.03 last 50.57)
 - open/close 2278 (time 1785394404.53 total 50.06 last 53.41)
 - open/close 2688 (time 1785394414.55 total 60.08 last 40.94)
 - open/close 3268 (time 1785394424.55 total 70.09 last 57.95)
 - open/close 4107 (time 1785394434.56 total 80.09 last 83.85)
 - open/close 4837 (time 1785394444.57 total 90.10 last 72.97)
 - open/close 5436 (time 1785394454.57 total 100.10 last 59.90)
 - open/close 6048 (time 1785394464.60 total 110.14 last 60.97)
 - open/close 6748 (time 1785394474.61 total 120.14 last 69.96)
 - open/close 7589 (time 1785394484.62 total 130.15 last 84.03)
 - open/close 8254 (time 1785394494.62 total 140.15 last 66.47)
 - open/close 8929 (time 1785394504.63 total 150.16 last 67.44)
 - open/close 9716 (time 1785394514.63 total 160.17 last 78.67)
total: 10000 open/close in 165.10 seconds: 60.57 ops/second
 - unlinked 0 (time 1785394524 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
 - open/close 510 (time 1785394653.64 total 10.00 last 51.00)
 - open/close 923 (time 1785394663.64 total 20.00 last 41.29)
 - open/close 1495 (time 1785394673.65 total 30.01 last 57.18)
 - open/close 2049 (time 1785394683.65 total 40.01 last 55.40)
 - open/close 2731 (time 1785394693.65 total 50.01 last 68.19)
 - open/close 3372 (time 1785394703.65 total 60.01 last 64.08)
 - open/close 4004 (time 1785394713.67 total 70.03 last 63.11)
 - open/close 4651 (time 1785394723.67 total 80.03 last 64.66)
 - open/close 5501 (time 1785394733.67 total 90.04 last 84.97)
 - open/close 6259 (time 1785394743.68 total 100.04 last 75.77)
 - open/close 6946 (time 1785394753.69 total 110.05 last 68.60)
 - open/close 7497 (time 1785394763.70 total 120.06 last 55.08)
 - open/close 8007 (time 1785394773.72 total 130.08 last 50.87)
 - open/close 8683 (time 1785394783.73 total 140.09 last 67.52)
 - open/close 9351 (time 1785394793.77 total 150.13 last 66.58)
 - open/close 9957 (time 1785394803.78 total 160.14 last 60.54)
total: 10000 open/close in 161.02 seconds: 62.10 ops/second
 - unlinked 0 (time 1785394810 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 531 (time 1785394919.50 total 10.02 last 53.00)
 - open/close 869 (time 1785394929.50 total 20.02 last 33.79)
 - open/close 1392 (time 1785394939.50 total 30.02 last 52.29)
 - open/close 1971 (time 1785394949.52 total 40.04 last 57.83)
 - open/close 2636 (time 1785394959.54 total 50.06 last 66.37)
 - open/close 3201 (time 1785394969.54 total 60.06 last 56.46)
 - open/close 3776 (time 1785394979.57 total 70.09 last 57.34)
 - open/close 4193 (time 1785394989.59 total 80.11 last 41.61)
 - open/close 4712 (time 1785394999.61 total 90.13 last 51.82)
 - open/close 5290 (time 1785395009.61 total 100.13 last 57.77)
 - open/close 5630 (time 1785395019.62 total 110.14 last 33.98)
 - open/close 6124 (time 1785395029.63 total 120.15 last 49.37)
 - open/close 6738 (time 1785395039.63 total 130.15 last 61.39)
 - open/close 7426 (time 1785395049.69 total 140.21 last 68.36)
 - open/close 8426 (time 1785395059.71 total 150.23 last 99.80)
 - open/close 9108 (time 1785395069.74 total 160.26 last 68.02)
 - open/close 9661 (time 1785395079.76 total 170.28 last 55.19)
total: 10000 open/close in 175.35 seconds: 57.03 ops/second
 - unlinked 0 (time 1785395089 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 405 (time 1785395209.44 total 10.00 last 40.49)
 - open/close 887 (time 1785395219.44 total 20.01 last 48.17)
 - open/close 1769 (time 1785395229.45 total 30.02 last 88.10)
 - open/close 2605 (time 1785395239.46 total 40.03 last 83.55)
 - open/close 3201 (time 1785395249.47 total 50.04 last 59.52)
 - open/close 3657 (time 1785395259.49 total 60.06 last 45.51)
 - open/close 4022 (time 1785395269.55 total 70.12 last 36.29)
 - open/close 4439 (time 1785395279.67 total 80.24 last 41.20)
 - open/close 4813 (time 1785395289.67 total 90.24 last 37.40)
 - open/close 5523 (time 1785395299.70 total 100.27 last 70.82)
 - open/close 6207 (time 1785395309.71 total 110.28 last 68.31)
 - open/close 6700 (time 1785395319.74 total 120.30 last 49.17)
 - open/close 7306 (time 1785395329.75 total 130.32 last 60.52)
 - open/close 7744 (time 1785395339.77 total 140.33 last 43.73)
 - open/close 8162 (time 1785395349.77 total 150.34 last 41.78)
 - open/close 8804 (time 1785395359.80 total 160.37 last 64.01)
 - open/close 9308 (time 1785395369.82 total 170.38 last 50.33)
 - open/close 9775 (time 1785395379.82 total 180.39 last 46.68)
total: 10000 open/close in 185.07 seconds: 54.03 ops/second
 - unlinked 0 (time 1785395393 ; total 0 ; last 0)
total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second
 - open/close 822 (time 1785395531.12 total 10.01 last 82.14)
 - open/close 1446 (time 1785395541.13 total 20.02 last 62.34)
 - open/close 2253 (time 1785395551.14 total 30.03 last 80.63)
 - open/close 2897 (time 1785395561.15 total 40.04 last 64.31)
 - open/close 3370 (time 1785395571.16 total 50.05 last 47.26)
 - open/close 3998 (time 1785395581.17 total 60.06 last 62.76)
 - open/close 4482 (time 1785395591.18 total 70.07 last 48.33)
 - open/close 5011 (time 1785395601.18 total 80.07 last 52.89)
 - open/close 5532 (time 1785395611.19 total 90.07 last 52.09)
 - open/close 6044 (time 1785395621.19 total 100.08 last 51.16)
 - open/close 6855 (time 1785395631.20 total 110.08 last 81.08)
 - open/close 7282 (time 1785395641.22 total 120.10 last 42.62)
 - open/close 7581 (time 1785395651.27 total 130.16 last 29.73)
 - open/close 8175 (time 1785395661.29 total 140.18 last 59.31)
 - open/close 8643 (time 1785395671.30 total 150.18 last 46.76)
 - open/close 9560 (time 1785395681.30 total 160.19 last 91.64)
total: 10000 open/close in 169.72 seconds: 58.92 ops/second
 - unlinked 0 (time 1785395697 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 427 (time 1785395821.73 total 10.01 last 42.67)
 - open/close 784 (time 1785395831.74 total 20.02 last 35.67)
 - open/close 1259 (time 1785395841.76 total 30.04 last 47.41)
 - open/close 1767 (time 1785395851.77 total 40.05 last 50.75)
 - open/close 2279 (time 1785395861.77 total 50.05 last 51.19)
 - open/close 2729 (time 1785395871.82 total 60.09 last 44.78)
 - open/close 3115 (time 1785395881.83 total 70.11 last 38.56)
 - open/close 3539 (time 1785395891.83 total 80.11 last 42.38)
 - open/close 4060 (time 1785395901.84 total 90.12 last 52.06)
 - open/close 4695 (time 1785395911.85 total 100.13 last 63.45)
 - open/close 5349 (time 1785395921.86 total 110.13 last 65.34)
 - open/close 6198 (time 1785395931.86 total 120.13 last 84.89)
 - open/close 7067 (time 1785395941.87 total 130.15 last 86.80)
 - open/close 7647 (time 1785395951.88 total 140.16 last 57.94)
 - open/close 8237 (time 1785395961.89 total 150.17 last 58.93)
 - open/close 8613 (time 1785395971.92 total 160.20 last 37.50)
 - open/close 8955 (time 1785395981.93 total 170.20 last 34.17)
 - open/close 9354 (time 1785395991.95 total 180.23 last 39.80)
 - open/close 9670 (time 1785396001.95 total 190.23 last 31.60)
 - open/close 9992 (time 1785396011.95 total 200.23 last 32.20)
total: 10000 open/close in 200.45 seconds: 49.89 ops/second
 - unlinked 0 (time 1785396018 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 628 (time 1785396135.37 total 10.00 last 62.79)
 - open/close 1173 (time 1785396145.38 total 20.02 last 54.43)
 - open/close 1658 (time 1785396155.39 total 30.02 last 48.47)
 - open/close 2151 (time 1785396165.41 total 40.05 last 49.17)
 - open/close 2673 (time 1785396175.45 total 50.08 last 52.03)
 - open/close 3165 (time 1785396185.46 total 60.10 last 49.13)
 - open/close 3551 (time 1785396195.46 total 70.10 last 38.59)
 - open/close 4022 (time 1785396205.47 total 80.10 last 47.09)
 - open/close 4481 (time 1785396215.47 total 90.10 last 45.89)
 - open/close 4817 (time 1785396225.50 total 100.13 last 33.51)
 - open/close 5255 (time 1785396235.54 total 110.17 last 43.61)
 - open/close 5640 (time 1785396245.58 total 120.22 last 38.34)
 - open/close 6103 (time 1785396255.59 total 130.22 last 46.27)
 - open/close 6695 (time 1785396265.59 total 140.22 last 59.19)
 - open/close 7346 (time 1785396275.60 total 150.24 last 65.00)
 - open/close 7932 (time 1785396285.61 total 160.24 last 58.59)
 - open/close 8605 (time 1785396295.61 total 170.24 last 67.29)
 - open/close 9502 (time 1785396305.61 total 180.25 last 89.68)
total: 10000 open/close in 185.31 seconds: 53.96 ops/second
 - unlinked 0 (time 1785396316 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 848 (time 1785396421.19 total 10.02 last 84.65)
 - open/close 1234 (time 1785396431.19 total 20.02 last 38.60)
 - open/close 1964 (time 1785396441.20 total 30.03 last 72.90)
 - open/close 2542 (time 1785396451.20 total 40.03 last 57.79)
 - open/close 3345 (time 1785396461.20 total 50.04 last 80.29)
 - open/close 4011 (time 1785396471.21 total 60.04 last 66.57)
 - open/close 4533 (time 1785396481.22 total 70.06 last 52.12)
 - open/close 4979 (time 1785396491.23 total 80.06 last 44.58)
 - open/close 5521 (time 1785396501.25 total 90.08 last 54.10)
 - open/close 6135 (time 1785396511.25 total 100.09 last 61.36)
 - open/close 6847 (time 1785396521.26 total 110.09 last 71.19)
 - open/close 7399 (time 1785396531.27 total 120.10 last 55.12)
 - open/close 7986 (time 1785396541.28 total 130.12 last 58.62)
 - open/close 8316 (time 1785396551.33 total 140.16 last 32.86)
 - open/close 8707 (time 1785396561.33 total 150.16 last 39.09)
 - open/close 9243 (time 1785396571.35 total 160.18 last 53.47)
 - open/close 9865 (time 1785396581.36 total 170.19 last 62.18)
total: 10000 open/close in 172.87 seconds: 57.85 ops/second
 - unlinked 0 (time 1785396589 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg212-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg212-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg212-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1
Started lustre-OST0000
oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg212-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg212-client.virtnet:  -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre
On OST0, 9790 used inodes rc=0
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg212-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg212-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.112@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.112@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.112@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg212-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1
Started lustre-OST0000
oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg212-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg212-client.virtnet:  -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg212-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg212-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server
stop mds service on oleg212-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server

unloading modules via unload_modules_local on: 'oleg212-server'
oleg212-server: oleg212-server.virtnet: executing unload_modules_local
oleg212-server: 
modules unloaded.
pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2
pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2
pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2